Architecting Data and Machine Learning Platforms

Chapter 7. Converging to a Lakehouse

As you know by this point, there are two main approaches that organizations can take when designing their data platform: following a data lake or a DWH paradigm. Both approaches come with pros and cons, but the question is: is it possible to make both technologies coexist to have a convergent architecture? In this chapter we will explore this topic, starting with a brief motivation for this idea, and then we will analyze two broad variants of the convergent architecture—known as the lakehouse architecture—and help you decide how to choose between them.

The lakehouse concept is becoming increasingly popular because it allows for a more flexible and scalable way to store and analyze structured, semistructured, and unstructured data at scale. A lakehouse can handle the entire lifecycle of structured and unstructured data, combining the best of the data lake and DWH approaches you learned in the previous two chapters in a governed manner. At the end of this chapter we will describe how to evolve toward the lakehouse architecture.

The Need for a Unique Architecture

Data lakes and DWHs emerged to meet the needs of different users. An organization with both types of users is faced with an unappealing choice.

User Personas

As you have learned in the previous chapters, some key differences between a data lake and a DWH are related to the type of data that can be ingested and the ability to land unprocessed (raw) data into a common location. Therefore, ...