Building Medallion Architectures

Chapter 2. Demystifying the Medallion: Practical Insights into the Layers

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

In Chapter 1, we introduced the Medallion Architecture, a data design pattern used to logically organize data within a modern lakehouse. This pattern utilizes three layers (Bronze, Silver, and Gold) to progressively refine datasets throughout the life cycles of data ingestion, data transformation and data loading into various destinations. Explaining the Medallion Architecture to organizations often feels like opening a can of worms, as each layer, meant to address different concerns, lacks clear definitions and descriptive guidelines. This ambiguity leads to more questions than answers, creating a cycle of confusion and inefficiency.

The Three-Layered Design

Despite the popularity of this three-layered design, there’s significant debate about the scope, purpose, and best practices for each layer. Moreover, the gap between theory and practical application ...