book

Deciphering Data Architectures

by James Serra

February 2024

Intermediate to advanced

278 pages

7h 29m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is Big Data, and How Can It Help You?Data MaturityStage 1: ReactiveStage 2: InformativeStage 3: PredictiveStage 4: TransformativeSelf-Service Business IntelligenceSummary
Evolution of Data ArchitecturesRelational Data WarehouseData LakeModern Data WarehouseData FabricData LakehouseData MeshSummary
What Is an ADS?Why Hold an ADS?Before the ADSPreparingInviting ParticipantsConducting the ADSIntroductionsDiscoveryWhiteboardingAfter the ADSTips for Conducting an ADSSummary
What Is a Relational Data Warehouse?What a Data Warehouse Is NotThe Top-Down ApproachWhy Use a Relational Data Warehouse?Drawbacks to Using a Relational Data WarehousePopulating a Data WarehouseHow Often to Extract the DataExtraction MethodsHow to Determine What Data Has Changed Since the Last ExtractionThe Death of the Relational Data Warehouse Has Been Greatly ExaggeratedSummary
What Is a Data Lake?Why Use a Data Lake?Bottom-Up ApproachBest Practices for Data Lake DesignMultiple Data LakesAdvantagesDisadvantagesSummary
Data Storage SolutionsData MartsOperational Data StoresData HubsData ProcessesMaster Data ManagementData Virtualization and Data FederationData CatalogsData MarketplacesSummary

Online Transaction Processing Versus Online Analytical ProcessingOperational and Analytical DataSymmetric Multiprocessing and Massively Parallel ProcessingLambda ArchitectureKappa ArchitecturePolyglot Persistence and Polyglot Data StoresSummary
Relational ModelingKeysEntity–Relationship DiagramsNormalization Rules and FormsTracking ChangesDimensional ModelingFacts, Dimensions, and KeysTracking ChangesDenormalizationCommon Data ModelData VaultThe Kimball and Inmon Data Warehousing MethodologiesInmon’s Top-Down MethodologyKimball’s Bottom-Up MethodologyChoosing a MethodologyHybrid ModelsMethodology MythsSummary
ETL Versus ELTReverse ETLBatch Processing Versus Real-Time ProcessingBatch Processing Pros and ConsReal-Time Processing Pros and ConsData GovernanceSummary
The MDW ArchitecturePros and Cons of the MDW ArchitectureCombining the RDW and Data LakeData LakeRelational Data WarehouseStepping Stones to the MDWEDW AugmentationTemporary Data Lake Plus EDWAll-in-OneCase Study: Wilson & Gunkerk’s Strategic Shift to an MDWChallengeSolutionOutcomeSummary
The Data Fabric ArchitectureData Access PoliciesMetadata CatalogMaster Data ManagementData VirtualizationReal-Time ProcessingAPIsServicesProductsWhy Transition from an MDW to a Data Fabric Architecture?Potential DrawbacksSummary
Delta Lake FeaturesPerformance ImprovementsThe Data Lakehouse ArchitectureWhat If You Skip the Relational Data Warehouse?Relational Serving LayerSummary
A Decentralized Data ArchitectureData Mesh HypeDehghani’s Four Principles of Data MeshPrinciple #1: Domain OwnershipPrinciple #2: Data as a ProductPrinciple #3: Self-Serve Data Infrastructure as a PlatformPrinciple #4: Federated Computational GovernanceThe “Pure” Data MeshData DomainsData Mesh Logical ArchitectureDifferent TopologiesData Mesh Versus Data FabricUse CasesSummary
MythsMyth: Using Data Mesh Is a Silver Bullet That Solves All Data Challenges QuicklyMyth: A Data Mesh Will Replace Your Data Lake and Data WarehouseMyth: Data Warehouse Projects Are All Failing, and a Data Mesh Will Solve That ProblemMyth: Building a Data Mesh Means Decentralizing Absolutely EverythingMyth: You Can Use Data Virtualization to Create a Data MeshConcernsPhilosophical and Conceptual MattersCombining Data in a Decentralized EnvironmentOther Issues of DecentralizationComplexityDuplicationFeasibilityPeopleDomain-Level BarriersOrganizational Assessment: Should You Adopt a Data Mesh?Recommendations for Implementing a Successful Data MeshThe Future of Data MeshZooming Out: Understanding Data Architectures and Their ApplicationsSummary
Team Organization: Roles and ResponsibilitiesRoles for MDW, Data Fabric, or Data LakehouseRoles for Data MeshWhy Projects Fail: Pitfalls and PreventionPitfall: Allowing Executives to Think That BI Is “Easy”Pitfall: Using the Wrong TechnologiesPitfall: Gathering Too Many Business RequirementsPitfall: Gathering Too Few Business RequirementsPitfall: Presenting Reports Without Validating Their Contents FirstPitfall: Hiring an Inexperienced Consulting CompanyPitfall: Hiring a Consulting Company That Outsources Development to Offshore WorkersPitfall: Passing Project Ownership Off to ConsultantsPitfall: Neglecting the Need to Transfer Knowledge Back into the OrganizationPitfall: Slashing the Budget Midway Through the ProjectPitfall: Starting with an End Date and Working BackwardPitfall: Structuring the Data Warehouse to Reflect the Source Data Rather Than the Business’s NeedsPitfall: Presenting End Users with a Solution with Slow Response Times or Other Performance IssuesPitfall: Overdesigning (or Underdesigning) Your Data ArchitecturePitfall: Poor Communication Between IT and the Business DomainsTips for SuccessDon’t Skimp on Your InvestmentInvolve Users, Show Them Results, and Get Them ExcitedAdd Value to New Reports and DashboardsAsk End Users to Build a PrototypeFind a Project Champion/SponsorMake a Project Plan That Aims for 80% EfficiencySummary
Choosing a PlatformOpen Source SolutionsOn-Premises SolutionsCloud Provider SolutionsCloud Service ModelsMajor Cloud ProvidersMulti-Cloud SolutionsSoftware FrameworksHadoopDatabricksSnowflakeSummary

Content preview from Deciphering Data Architectures

Chapter 16. Technologies

As you delve into this chapter, your first crucial decision is choosing between open source solutions and cloud provider offerings. However, there’s more to consider. I’ll guide you in thinking about the scale of your data needs and your organization’s agility requirements.

Beyond this, we’ll explore the landscape of cloud service models, from major providers to the complexity of multi-cloud solutions. The chapter ends by introducing you to three essential software frameworks: Hadoop, Databricks, and Snowflake.

Your answers to the questions in this chapter will profoundly shape your data architecture. It’s not just open source versus proprietary; it’s about crafting the perfect technology mix to suit your unique needs in the world of data architecture.

Choosing a Platform

The decision to use open source software or software from a cloud service provider (CSP) depends on various factors, including specific requirements, budget, expertise, data sensitivity, and control preferences. Evaluate your organization’s needs carefully and consider the trade-offs between the advantages of CSP software and the flexibility and customization offered by open source software (OSS).

Open Source Solutions

Open source software refers to software that is distributed for free with its source code openly available for anyone to view, modify, and distribute. It is typically developed collaboratively by a community of programmers who contribute their knowledge and expertise ...