Initially, data lakes will not be well organized or maintained with a broader audience in mind. When first loading data sources to a data lake, their structure is pretty much the same as before they ended up in the lake (Figure 8.1). These structures can be hard to understand and query; however, they do not need to be cleaned up too much yet. The majority of the cleaning will take place in creating the data warehouse.
The areas of focus for data lake maintenance are around:
SQL
Extracting and loading of data sources
Performance
Figure 8.1 Source data being loaded with SQL into a data lake.
These maintenance activities can be expensive once data is extracted and loaded with custom scripts. It's necessary to have in‐depth knowledge of the data, such as where data originates, the details of the APIs, and the data structures inside it. Also, the potential need to write new code and maintain existing code when data sources update is high. We recommend avoiding manual extract and load; use tools like Fivetran or Stitch, which automatically handle data source updates so that any data engineers can focus on more critical tasks.
Why SQL?
While the data could be loaded into the lake in a variety of formats, we recommend SQL (Figure 8.1). It's the standard language for relational database management systems (which is what a Data Warehouse is ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month, and much more.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
Julian F.
Head of Cybersecurity
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.
Addison B.
Field Engineer
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.
Amir M.
Data Platform Tech Lead
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
