What Is a Data Lake?

Chapter 4. Architecting Multiple Data Lakes

Enterprises build data lakes for a variety of reasons. Some start as single-project data puddles created by a line-of-business or project team and grow into data lakes gradually. Some start as ETL offloading projects by IT and pick up additional users and analytic use cases along the way. Others are designed from the get-go as centralized data lakes by IT and analytic teams working together. Yet others are created in the cloud as shadow IT for business teams that don’t want to wait for the official IT team.

Similarly, enterprises don’t always have their data lakes in the same public cloud. Some data lakes contain data that enterprises are not allowed to keep or not comfortable with keeping in public clouds, or regulations require having different data lakes to comply with data privacy and other local laws. Sometimes data lakes are created by different divisions or are acquired by companies in different public clouds. Other data lakes are deliberately created in different public cloud providers to take advantage of unique capabilities, data locality, vendor independence, and a variety of other reasons.

Regardless of the reasons, many enterprises end up with multiple data lakes. The question then becomes, should these be merged into one or kept separate? This chapter answers this question first and then explores approaches to managing multiple data lakes by applying virtual constructs such as data federation, data fabrics, and data catalogs. ...