The Cloud Data Lake

Chapter 7. Decision Framework for Your Architecture

Truly successful decision-making relies on a balance between deliberate and instinctive thinking.

Malcolm Gladwell

With the content in the prior chapters, I’ve covered the fundamental concepts of designing the architecture of big data processing on the cloud, the different architecture choices for big data lakes, design considerations for your cloud data lake solution, and the tools you have available to optimize your solution for scale and performance. In this chapter, I will put all of these concepts together and elaborate on how this understanding serves you at different points on your data lake journey. In building as well as consuming these cloud data lake solutions, I have talked to hundreds of customers across various industries—financial services, retail, consumer goods, health care, manufacturing, and technology sectors. This chapter distills the experiences of all these customers into a core set of common patterns and thought processes reflecting different levels of organizational maturity, available skill sets, and current states of technology adoption.

Although each of these customers has its own arc of problems and requirements, the common thread for every one of them is that they realize the importance of data and how it can help inform and transform their businesses. Data is viewed as a fundamental component to maintain their competitive advantage, retain and grow their customer base, and drive efficiencies ...