book

Deciphering Data Architectures

by James Serra

February 2024

Intermediate to advanced

278 pages

7h 29m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is Big Data, and How Can It Help You?Data MaturityStage 1: ReactiveStage 2: InformativeStage 3: PredictiveStage 4: TransformativeSelf-Service Business IntelligenceSummary
Evolution of Data ArchitecturesRelational Data WarehouseData LakeModern Data WarehouseData FabricData LakehouseData MeshSummary
What Is an ADS?Why Hold an ADS?Before the ADSPreparingInviting ParticipantsConducting the ADSIntroductionsDiscoveryWhiteboardingAfter the ADSTips for Conducting an ADSSummary
What Is a Relational Data Warehouse?What a Data Warehouse Is NotThe Top-Down ApproachWhy Use a Relational Data Warehouse?Drawbacks to Using a Relational Data WarehousePopulating a Data WarehouseHow Often to Extract the DataExtraction MethodsHow to Determine What Data Has Changed Since the Last ExtractionThe Death of the Relational Data Warehouse Has Been Greatly ExaggeratedSummary
What Is a Data Lake?Why Use a Data Lake?Bottom-Up ApproachBest Practices for Data Lake DesignMultiple Data LakesAdvantagesDisadvantagesSummary
Data Storage SolutionsData MartsOperational Data StoresData HubsData ProcessesMaster Data ManagementData Virtualization and Data FederationData CatalogsData MarketplacesSummary

Online Transaction Processing Versus Online Analytical ProcessingOperational and Analytical DataSymmetric Multiprocessing and Massively Parallel ProcessingLambda ArchitectureKappa ArchitecturePolyglot Persistence and Polyglot Data StoresSummary
Relational ModelingKeysEntity–Relationship DiagramsNormalization Rules and FormsTracking ChangesDimensional ModelingFacts, Dimensions, and KeysTracking ChangesDenormalizationCommon Data ModelData VaultThe Kimball and Inmon Data Warehousing MethodologiesInmon’s Top-Down MethodologyKimball’s Bottom-Up MethodologyChoosing a MethodologyHybrid ModelsMethodology MythsSummary
ETL Versus ELTReverse ETLBatch Processing Versus Real-Time ProcessingBatch Processing Pros and ConsReal-Time Processing Pros and ConsData GovernanceSummary
The MDW ArchitecturePros and Cons of the MDW ArchitectureCombining the RDW and Data LakeData LakeRelational Data WarehouseStepping Stones to the MDWEDW AugmentationTemporary Data Lake Plus EDWAll-in-OneCase Study: Wilson & Gunkerk’s Strategic Shift to an MDWChallengeSolutionOutcomeSummary
The Data Fabric ArchitectureData Access PoliciesMetadata CatalogMaster Data ManagementData VirtualizationReal-Time ProcessingAPIsServicesProductsWhy Transition from an MDW to a Data Fabric Architecture?Potential DrawbacksSummary
Delta Lake FeaturesPerformance ImprovementsThe Data Lakehouse ArchitectureWhat If You Skip the Relational Data Warehouse?Relational Serving LayerSummary
A Decentralized Data ArchitectureData Mesh HypeDehghani’s Four Principles of Data MeshPrinciple #1: Domain OwnershipPrinciple #2: Data as a ProductPrinciple #3: Self-Serve Data Infrastructure as a PlatformPrinciple #4: Federated Computational GovernanceThe “Pure” Data MeshData DomainsData Mesh Logical ArchitectureDifferent TopologiesData Mesh Versus Data FabricUse CasesSummary
MythsMyth: Using Data Mesh Is a Silver Bullet That Solves All Data Challenges QuicklyMyth: A Data Mesh Will Replace Your Data Lake and Data WarehouseMyth: Data Warehouse Projects Are All Failing, and a Data Mesh Will Solve That ProblemMyth: Building a Data Mesh Means Decentralizing Absolutely EverythingMyth: You Can Use Data Virtualization to Create a Data MeshConcernsPhilosophical and Conceptual MattersCombining Data in a Decentralized EnvironmentOther Issues of DecentralizationComplexityDuplicationFeasibilityPeopleDomain-Level BarriersOrganizational Assessment: Should You Adopt a Data Mesh?Recommendations for Implementing a Successful Data MeshThe Future of Data MeshZooming Out: Understanding Data Architectures and Their ApplicationsSummary
Team Organization: Roles and ResponsibilitiesRoles for MDW, Data Fabric, or Data LakehouseRoles for Data MeshWhy Projects Fail: Pitfalls and PreventionPitfall: Allowing Executives to Think That BI Is “Easy”Pitfall: Using the Wrong TechnologiesPitfall: Gathering Too Many Business RequirementsPitfall: Gathering Too Few Business RequirementsPitfall: Presenting Reports Without Validating Their Contents FirstPitfall: Hiring an Inexperienced Consulting CompanyPitfall: Hiring a Consulting Company That Outsources Development to Offshore WorkersPitfall: Passing Project Ownership Off to ConsultantsPitfall: Neglecting the Need to Transfer Knowledge Back into the OrganizationPitfall: Slashing the Budget Midway Through the ProjectPitfall: Starting with an End Date and Working BackwardPitfall: Structuring the Data Warehouse to Reflect the Source Data Rather Than the Business’s NeedsPitfall: Presenting End Users with a Solution with Slow Response Times or Other Performance IssuesPitfall: Overdesigning (or Underdesigning) Your Data ArchitecturePitfall: Poor Communication Between IT and the Business DomainsTips for SuccessDon’t Skimp on Your InvestmentInvolve Users, Show Them Results, and Get Them ExcitedAdd Value to New Reports and DashboardsAsk End Users to Build a PrototypeFind a Project Champion/SponsorMake a Project Plan That Aims for 80% EfficiencySummary
Choosing a PlatformOpen Source SolutionsOn-Premises SolutionsCloud Provider SolutionsCloud Service ModelsMajor Cloud ProvidersMulti-Cloud SolutionsSoftware FrameworksHadoopDatabricksSnowflakeSummary

Content preview from Deciphering Data Architectures

Chapter 13. Data Mesh Foundation

A data mesh is a decentralized data architecture with four specific characteristics. First, it requires independent teams within desginated domains to own their analytical data. Second, in a data mesh, data is treated and served as a product to help the data consumer to discover, trust, and utilize it for whatever purpose they like. Third, it relies on automated infrastructure provisioning. And fourth, it uses governance to ensure that all the independent data products are secure and follow global rules.

Although the concepts that make up a data mesh are not new, Zhamak Dehghani, CEO and founder of Nextdata, deserves credit for coining the term data mesh and combining those concepts.¹ Although others have formed various opinions about what a data mesh is, this book bases its definition on Dehghani’s work, in particular her four data mesh principles, which I discuss in this chapter. It is very important to understand that data mesh is a concept, not a technology. It is all about an organizational and cultural shift within companies. The technology used to build a data mesh could follow the modern data warehouse, data fabric, or data lakehouse architecture—or domains could even follow different architectures.

As we delve into the nuanced world of data mesh architectures in this chapter and the next, I offer a road map to navigate the complex data mesh landscape grounded in the principle of decentralization. In this chapter, I dissect the decentralized ...