Building Medallion Architectures

Chapter 6. Build the Silver Layer

In Chapter 5, the foundational design and setup of the Bronze layer in the Medallion architecture, based on Oceanic Airlines’ reference architecture, was explored. Deployment and configuration were examined, enriching understanding with numerous code snippets and examples. You learned that crafting a queryable Bronze layer presents significant challenges due to the ever-evolving and complex nature of source systems.

Moving forward, this chapter builds on that foundation by advancing to the construction of the Silver layer, the subsequent tier in the Medallion architecture, where the objective is to refine, cleanse, and standardize the data. We’ll explore several critical areas in the following order:

• Ensuring data integrity through a metadata-driven approach

• Cleansing data to improve quality

• Transforming data into a denormalized model for ease of access

• Enriching data, potentially by incorporating master data management

• Implementing data historization to keep track of changes over time

• Focusing on optimization jobs

• Orchestrating data pipelines with Airflow to automate end-to-end processing

Like Chapter 5, this chapter guides you through essential configuration steps and coding practices. There will be hands-on activities, including metadata-driven validation, data cleansing, and historization tasks that you can directly implement in your project.

In this chapter, we’ll also discusses several data transformation and data quality ...