Accelerate Machine Learning with a Unified Analytics Architecture

Chapter 2. Evolution of the Data Lake and Data Warehouse

This report assumes a basic understanding of the data warehouse and data lake architectures, as well as their advantages and disadvantages. It is important, however, to understand the nature of the evolution of these two systems.

At its inception about a decade ago, the data lake was quite different from the data warehouse and claimed a myriad of benefits. In the Venn diagram of data lake and warehouse value-adds, the overlap was minimal, aside from the general storage and retrieval of data, as seen in Figure 2-1.

When the data lake first made waves into the data landscape, the data warehouse ran on expensive, proprietary technologies, was not particularly parallelizable across multiple machines, and handled strictly structured data. This was problematic, and the data lakes of the world—particularly Apache Hadoop—presented a compelling alternative. For example, the Hadoop MapReduce programming model (read file from data source A, do one processing step, write the data back to disk, read again, do next processing step, write to disk, repeat until data processing is completed, and then move to data source B) lead to the first real data lake that used the Hadoop Distributed File System (HDFS) for data storage.

Figure 2-1. Venn diagram of a historical data warehouse versus a historical data lake

The data lake broke ground by ...