Before building the single source of truth, we recommend first making idealized versions of each of the sources in the data lake. These are called staging schemas (Figure 10.1).
Generally, a staging schema holds models (i.e. tables or views) that handle transformations of this type. The tables and columns that are unneeded or empty are removed, weird names are improved, and unwanted data is filtered out. The value of investing time into cleaning and renaming data fields lies in making the data ready for more complex numeric transformations without needing to worry about these sorts of consistencies. Some may be concerned about the space implications from having many staging tables or repeated data. The good news, however, is that the compression strategies of a modern C‐Store warehouse handle exactly this kind of data organization without noticeable performance hits.
Figure 10.1 The four stages of agile data organization with an intermediary step that illustrates where a “staging schema” is relevant.
Here's a last note on process: as each source is cleaned, it never hurts to give read‐only access to departments across your organization. Showing the value of the modeling and asking for feedback during creation are ways your team can earn trust and confidence from stakeholders.
Orient to the Schemas
Go through the list of tables in the schemas ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month, and much more.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
Julian F.
Head of Cybersecurity
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.
Addison B.
Field Engineer
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.
Amir M.
Data Platform Tech Lead
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
