book

Deciphering Data Architectures

by James Serra

February 2024

Intermediate to advanced

278 pages

7h 29m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is Big Data, and How Can It Help You?Data MaturityStage 1: ReactiveStage 2: InformativeStage 3: PredictiveStage 4: TransformativeSelf-Service Business IntelligenceSummary
Evolution of Data ArchitecturesRelational Data WarehouseData LakeModern Data WarehouseData FabricData LakehouseData MeshSummary
What Is an ADS?Why Hold an ADS?Before the ADSPreparingInviting ParticipantsConducting the ADSIntroductionsDiscoveryWhiteboardingAfter the ADSTips for Conducting an ADSSummary
What Is a Relational Data Warehouse?What a Data Warehouse Is NotThe Top-Down ApproachWhy Use a Relational Data Warehouse?Drawbacks to Using a Relational Data WarehousePopulating a Data WarehouseHow Often to Extract the DataExtraction MethodsHow to Determine What Data Has Changed Since the Last ExtractionThe Death of the Relational Data Warehouse Has Been Greatly ExaggeratedSummary
What Is a Data Lake?Why Use a Data Lake?Bottom-Up ApproachBest Practices for Data Lake DesignMultiple Data LakesAdvantagesDisadvantagesSummary
Data Storage SolutionsData MartsOperational Data StoresData HubsData ProcessesMaster Data ManagementData Virtualization and Data FederationData CatalogsData MarketplacesSummary

Online Transaction Processing Versus Online Analytical ProcessingOperational and Analytical DataSymmetric Multiprocessing and Massively Parallel ProcessingLambda ArchitectureKappa ArchitecturePolyglot Persistence and Polyglot Data StoresSummary
Relational ModelingKeysEntity–Relationship DiagramsNormalization Rules and FormsTracking ChangesDimensional ModelingFacts, Dimensions, and KeysTracking ChangesDenormalizationCommon Data ModelData VaultThe Kimball and Inmon Data Warehousing MethodologiesInmon’s Top-Down MethodologyKimball’s Bottom-Up MethodologyChoosing a MethodologyHybrid ModelsMethodology MythsSummary
ETL Versus ELTReverse ETLBatch Processing Versus Real-Time ProcessingBatch Processing Pros and ConsReal-Time Processing Pros and ConsData GovernanceSummary
The MDW ArchitecturePros and Cons of the MDW ArchitectureCombining the RDW and Data LakeData LakeRelational Data WarehouseStepping Stones to the MDWEDW AugmentationTemporary Data Lake Plus EDWAll-in-OneCase Study: Wilson & Gunkerk’s Strategic Shift to an MDWChallengeSolutionOutcomeSummary
The Data Fabric ArchitectureData Access PoliciesMetadata CatalogMaster Data ManagementData VirtualizationReal-Time ProcessingAPIsServicesProductsWhy Transition from an MDW to a Data Fabric Architecture?Potential DrawbacksSummary
Delta Lake FeaturesPerformance ImprovementsThe Data Lakehouse ArchitectureWhat If You Skip the Relational Data Warehouse?Relational Serving LayerSummary
A Decentralized Data ArchitectureData Mesh HypeDehghani’s Four Principles of Data MeshPrinciple #1: Domain OwnershipPrinciple #2: Data as a ProductPrinciple #3: Self-Serve Data Infrastructure as a PlatformPrinciple #4: Federated Computational GovernanceThe “Pure” Data MeshData DomainsData Mesh Logical ArchitectureDifferent TopologiesData Mesh Versus Data FabricUse CasesSummary
MythsMyth: Using Data Mesh Is a Silver Bullet That Solves All Data Challenges QuicklyMyth: A Data Mesh Will Replace Your Data Lake and Data WarehouseMyth: Data Warehouse Projects Are All Failing, and a Data Mesh Will Solve That ProblemMyth: Building a Data Mesh Means Decentralizing Absolutely EverythingMyth: You Can Use Data Virtualization to Create a Data MeshConcernsPhilosophical and Conceptual MattersCombining Data in a Decentralized EnvironmentOther Issues of DecentralizationComplexityDuplicationFeasibilityPeopleDomain-Level BarriersOrganizational Assessment: Should You Adopt a Data Mesh?Recommendations for Implementing a Successful Data MeshThe Future of Data MeshZooming Out: Understanding Data Architectures and Their ApplicationsSummary
Team Organization: Roles and ResponsibilitiesRoles for MDW, Data Fabric, or Data LakehouseRoles for Data MeshWhy Projects Fail: Pitfalls and PreventionPitfall: Allowing Executives to Think That BI Is “Easy”Pitfall: Using the Wrong TechnologiesPitfall: Gathering Too Many Business RequirementsPitfall: Gathering Too Few Business RequirementsPitfall: Presenting Reports Without Validating Their Contents FirstPitfall: Hiring an Inexperienced Consulting CompanyPitfall: Hiring a Consulting Company That Outsources Development to Offshore WorkersPitfall: Passing Project Ownership Off to ConsultantsPitfall: Neglecting the Need to Transfer Knowledge Back into the OrganizationPitfall: Slashing the Budget Midway Through the ProjectPitfall: Starting with an End Date and Working BackwardPitfall: Structuring the Data Warehouse to Reflect the Source Data Rather Than the Business’s NeedsPitfall: Presenting End Users with a Solution with Slow Response Times or Other Performance IssuesPitfall: Overdesigning (or Underdesigning) Your Data ArchitecturePitfall: Poor Communication Between IT and the Business DomainsTips for SuccessDon’t Skimp on Your InvestmentInvolve Users, Show Them Results, and Get Them ExcitedAdd Value to New Reports and DashboardsAsk End Users to Build a PrototypeFind a Project Champion/SponsorMake a Project Plan That Aims for 80% EfficiencySummary
Choosing a PlatformOpen Source SolutionsOn-Premises SolutionsCloud Provider SolutionsCloud Service ModelsMajor Cloud ProvidersMulti-Cloud SolutionsSoftware FrameworksHadoopDatabricksSnowflakeSummary

Content preview from Deciphering Data Architectures

Part III. Data Architectures

Now that you have a good understanding of the data architecture concepts, it’s time to get into the meat of the book. The five chapters of Part III cover four architectures, in order of when they first appeared: the modern data warehouse, data fabric, data lakehouse, and data mesh.

These are my interpretations of the data architectures; others may have different opinions. In the end, your solution may even be a combination of these architectures.