Architecting Data and Machine Learning Platforms

Chapter 5. Architecting a Data Lake

A data lake is the part of the data platform that captures raw, ungoverned data from across an organization and supports compute tools from the Apache ecosystem. In this chapter, we will go into more detail about this concept, which is important when designing modern data platforms. The cloud can provide a boost to the different use cases that can be implemented on top of it, as you will read throughout the chapter.

We will start with a recap of why you might want to store raw, ungoverned data that only supports basic compute. Then, we discuss architecture design and implementation details in the cloud. Even though data lakes were originally intended only for basic data processing, it is now possible to democratize data access and reporting using just a data lake—because of integrations with other solutions through APIs and connectors, the data within a data lake can be made much more fit for purpose. We will finally take a bird’s-eye perspective on a very common way to speed up analysis and experimentation with data within an organization by leveraging data science notebooks.

Data Lake and the Cloud—A Perfect Marriage

Data helps organizations make better decisions, faster. It’s the center of everything from applications to security, and more data means more need for processing power, which cloud solutions can provide.

Challenges with On-Premises Data Lakes

Organizations need a place to store all types of data, including unstructured data ...