book

Building Medallion Architectures

by Piethein Strengholt

March 2025

Intermediate to advanced

396 pages

11h 36m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Who Should Read This BookNavigating This BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is a Medallion Architecture?A Brief History of Data Warehouse ArchitectureOLTP SystemsData WarehousesThe Staging AreaInmon MethodologyKimball MethodologyKey Takeaways from Traditional Data WarehousesA Brief History of Data LakesHadoop’s Distributed File SystemMapReduceApache HiveSpark ProjectMoving Forward with Data LakesA Brief History of Lakehouse ArchitectureFounders of SparkEmergence of Open Table FormatsThe Rise of Lakehouse ArchitecturesMedallion Architecture and Its Practical ChallengesConclusion
Foundational PreconditionsExtra Landing ZonesRaw DataBatch ProcessingReal-Time Data ProcessingSpark Structured StreamingChange Data FeedChange Data CaptureConsiderations and Learning ResourcesETL and Orchestration ToolsManaging Delta TablesZ-OrderingV-OrderingTable PartitioningLiquid ClusteringCompaction and Optimized WritesDeltaLogConclusion
The Three-Layered DesignBronze LayerProcessing HierarchyProcessing Full Data LoadsProcessing Incremental Data LoadsData Historization Within the Bronze LayerSchema Evolution and ManagementMergeSchema and Schema EnforcementTechnical Validation ChecksUsage and GovernanceThe Bronze Layer in PracticeSilver LayerCleaning Data ActivitiesDesigning the Silver Layer’s Data ModelHarmonization with Other Sources3NF and Data VaultOperational Querying and Machine LearningManaging Overlapping RequirementsAutomation TasksThe Silver Layer in PracticeGold LayerStar SchemaStar Schema Design NuancesCurated, Semantic, and Platinum LayersOne-Big-Table DesignServing LayerThe Gold Layer in PracticeConclusion
Our Case Study: Oceanic AirlinesIntroducing Microsoft FabricDomainsWorkspaces and CapacitiesOneLakeData Engineering with SparkData Warehousing with T-SQLOther Fabric Workload TypesSetting Up the FoundationSetting up CapacitiesSetting up DomainsSetting up WorkspacesCreating LakehousesCapacity ConsiderationsDomain ConsiderationsWorkspace ConsiderationsLakehouse Entities ConsiderationsStorage Account ConsiderationsConclusion
Building the Data PipelineDeploying the AdventureWorks Sample DatabaseSet Up an Azure SQL Database ConnectionCreating a New Data PipelineAdditional ConsiderationsImplementation of Lakehouse TablesTraverse Parquet Files to Managed Delta TablesUsing External TablesUpdating Tables with MERGE OperationsSpark Structured StreamingUsing Change Data CaptureNavigating Data Handling TechniquesSchema ManagementCreate Tables Without Defining SchemasDefine Schemas with the DataFrame APISQL DDL StatementsYAML or JSON ConfigurationsMetadata-Driven ApproachDatabricks Auto LoaderThird-Party ToolsHandling Schema EvolutionConclusion
Quick RecapImplementation of a Metadata-Driven ApproachImplementation of the Metadata StoreImplementation of Dynamic Data ValidationsImprovement AreasData CleansingImplementation of Data Cleansing TasksData Cleansing ConsiderationsData Transformation Frameworks and Data Quality ToolsOptimization of Query Performance with DenormalizationLightweight EnrichmentsData HistorizationOptimization JobsOrchestration with Apache AirFlowFinal RecommendationsSilver-Layer Data as a ProductConclusion

Design of the Gold LayerTransform Data Using a Star SchemaCreation of the Semantic ModelCreation of the First Power BI ReportCreation of Task FlowsEnhancements for Gold-Layer DesignMicrosoft Fabric in PracticeData ProductsData Governance with Microsoft PurviewMicrosoft Purview Design ConsiderationsGuidance for Medallion ArchitecturesConclusion
Medallion ArchitectureOther ConsiderationsFinal Recommendations
Medallion ArchitectureFinOpsData ModelsData ContractsData Governance
Data Platform EvolutionMedallion ArchitectureData Products and SharingRecommendations and Best Practices
Decentralization of Data ManagementFlexibility in FederationMedallion MeshNumber of Medallion ArchitecturesMedallion Inner Architecture VariationsSeparate Data Product LayersTailored Medallions ArchitecturesAdaptability of the Bronze LayerSilver Layer VariationsGold Layer VariationsEnterprise Data ModelsMaster Data ManagementReference Data ManagementConclusion
Data GovernanceGovernance Within a Medallion ArchitectureUnity CatalogMedallion Architecture with Unity CatalogData ContractsContracts Within a CatalogContracts Within a MetastoreData Contracts Using YAML Files and GitOpsOther Data Contract SpecificationsData Security and Access ManagementConclusion
Unstructured Data ProcessingRetrieval-Augmented GenerationBronze LayerSilver LayerGold LayerIntegration of LLMs and Medallion ArchitecturesRole of AgentsTraining and Fine-Tuning LLMsFuture of Medallion ArchitecturesConclusion

Content preview from Building Medallion Architectures

Chapter 10. Case Study: Strategic Data Transformation at ZEISS

In this chapter, we explore the data strategies and infrastructure at ZEISS, a leader in the optics industry. We engage with Markus Morgner, head of the Enterprise Data Foundation; Sascha Saumer, a senior data engineer; and Gert Christen, who manages the Microsoft BI platform. These members of the ZEISS data team offer insights into the evolution of their data systems, strategic shifts, and the implementation of innovative technologies.

The discussion begins with an examination of the significant shift towards a centralized data strategy initiated in 2021. This strategic pivot aimed to consolidate scattered data practices into a streamlined, cohesive model, enhancing efficiency across the company’s various segments. Markus Morgner details the transition from a decentralized setup to a federated model that not only strengthens core control but also empowers individual segments with greater autonomy.

Further, Sascha Saumer and Gert Christen delve into the technical aspects of ZEISS’s data platforms, highlighting the transition to cloud technologies and the adoption of the data mesh concept. They explain how the company tailors its data handling to meet diverse business needs through a combination of centralized and decentralized approaches.

This chapter not only sheds light on the technicalities of managing large-scale data systems but also reflects on the broader implications for data governance, integration, and sharing ...