book

Building Medallion Architectures

by Piethein Strengholt

March 2025

Intermediate to advanced

396 pages

11h 36m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Who Should Read This BookNavigating This BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is a Medallion Architecture?A Brief History of Data Warehouse ArchitectureOLTP SystemsData WarehousesThe Staging AreaInmon MethodologyKimball MethodologyKey Takeaways from Traditional Data WarehousesA Brief History of Data LakesHadoop’s Distributed File SystemMapReduceApache HiveSpark ProjectMoving Forward with Data LakesA Brief History of Lakehouse ArchitectureFounders of SparkEmergence of Open Table FormatsThe Rise of Lakehouse ArchitecturesMedallion Architecture and Its Practical ChallengesConclusion
Foundational PreconditionsExtra Landing ZonesRaw DataBatch ProcessingReal-Time Data ProcessingSpark Structured StreamingChange Data FeedChange Data CaptureConsiderations and Learning ResourcesETL and Orchestration ToolsManaging Delta TablesZ-OrderingV-OrderingTable PartitioningLiquid ClusteringCompaction and Optimized WritesDeltaLogConclusion
The Three-Layered DesignBronze LayerProcessing HierarchyProcessing Full Data LoadsProcessing Incremental Data LoadsData Historization Within the Bronze LayerSchema Evolution and ManagementMergeSchema and Schema EnforcementTechnical Validation ChecksUsage and GovernanceThe Bronze Layer in PracticeSilver LayerCleaning Data ActivitiesDesigning the Silver Layer’s Data ModelHarmonization with Other Sources3NF and Data VaultOperational Querying and Machine LearningManaging Overlapping RequirementsAutomation TasksThe Silver Layer in PracticeGold LayerStar SchemaStar Schema Design NuancesCurated, Semantic, and Platinum LayersOne-Big-Table DesignServing LayerThe Gold Layer in PracticeConclusion
Our Case Study: Oceanic AirlinesIntroducing Microsoft FabricDomainsWorkspaces and CapacitiesOneLakeData Engineering with SparkData Warehousing with T-SQLOther Fabric Workload TypesSetting Up the FoundationSetting up CapacitiesSetting up DomainsSetting up WorkspacesCreating LakehousesCapacity ConsiderationsDomain ConsiderationsWorkspace ConsiderationsLakehouse Entities ConsiderationsStorage Account ConsiderationsConclusion
Building the Data PipelineDeploying the AdventureWorks Sample DatabaseSet Up an Azure SQL Database ConnectionCreating a New Data PipelineAdditional ConsiderationsImplementation of Lakehouse TablesTraverse Parquet Files to Managed Delta TablesUsing External TablesUpdating Tables with MERGE OperationsSpark Structured StreamingUsing Change Data CaptureNavigating Data Handling TechniquesSchema ManagementCreate Tables Without Defining SchemasDefine Schemas with the DataFrame APISQL DDL StatementsYAML or JSON ConfigurationsMetadata-Driven ApproachDatabricks Auto LoaderThird-Party ToolsHandling Schema EvolutionConclusion
Quick RecapImplementation of a Metadata-Driven ApproachImplementation of the Metadata StoreImplementation of Dynamic Data ValidationsImprovement AreasData CleansingImplementation of Data Cleansing TasksData Cleansing ConsiderationsData Transformation Frameworks and Data Quality ToolsOptimization of Query Performance with DenormalizationLightweight EnrichmentsData HistorizationOptimization JobsOrchestration with Apache AirFlowFinal RecommendationsSilver-Layer Data as a ProductConclusion

Design of the Gold LayerTransform Data Using a Star SchemaCreation of the Semantic ModelCreation of the First Power BI ReportCreation of Task FlowsEnhancements for Gold-Layer DesignMicrosoft Fabric in PracticeData ProductsData Governance with Microsoft PurviewMicrosoft Purview Design ConsiderationsGuidance for Medallion ArchitecturesConclusion
Medallion ArchitectureOther ConsiderationsFinal Recommendations
Medallion ArchitectureFinOpsData ModelsData ContractsData Governance
Data Platform EvolutionMedallion ArchitectureData Products and SharingRecommendations and Best Practices
Decentralization of Data ManagementFlexibility in FederationMedallion MeshNumber of Medallion ArchitecturesMedallion Inner Architecture VariationsSeparate Data Product LayersTailored Medallions ArchitecturesAdaptability of the Bronze LayerSilver Layer VariationsGold Layer VariationsEnterprise Data ModelsMaster Data ManagementReference Data ManagementConclusion
Data GovernanceGovernance Within a Medallion ArchitectureUnity CatalogMedallion Architecture with Unity CatalogData ContractsContracts Within a CatalogContracts Within a MetastoreData Contracts Using YAML Files and GitOpsOther Data Contract SpecificationsData Security and Access ManagementConclusion
Unstructured Data ProcessingRetrieval-Augmented GenerationBronze LayerSilver LayerGold LayerIntegration of LLMs and Medallion ArchitecturesRole of AgentsTraining and Fine-Tuning LLMsFuture of Medallion ArchitecturesConclusion

Content preview from Building Medallion Architectures

About the Author

Piethein Strengholt is a seasoned expert in data management with significant experience in chief data officer (CDO) and chief data architect roles. He has a strong track record of collaborating with CDO executives at large enterprises, where he focuses on driving community growth and aligning strategies with business goals. Piethein is also a prolific blogger and a sought-after speaker who regularly addresses the latest trends in data management, including data mesh concepts, data governance, and scaling strategies. He resides in the Netherlands with his family.