book

Deciphering Data Architectures

by James Serra

February 2024

Intermediate to advanced

278 pages

7h 29m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is Big Data, and How Can It Help You?Data MaturityStage 1: ReactiveStage 2: InformativeStage 3: PredictiveStage 4: TransformativeSelf-Service Business IntelligenceSummary
Evolution of Data ArchitecturesRelational Data WarehouseData LakeModern Data WarehouseData FabricData LakehouseData MeshSummary
What Is an ADS?Why Hold an ADS?Before the ADSPreparingInviting ParticipantsConducting the ADSIntroductionsDiscoveryWhiteboardingAfter the ADSTips for Conducting an ADSSummary
What Is a Relational Data Warehouse?What a Data Warehouse Is NotThe Top-Down ApproachWhy Use a Relational Data Warehouse?Drawbacks to Using a Relational Data WarehousePopulating a Data WarehouseHow Often to Extract the DataExtraction MethodsHow to Determine What Data Has Changed Since the Last ExtractionThe Death of the Relational Data Warehouse Has Been Greatly ExaggeratedSummary
What Is a Data Lake?Why Use a Data Lake?Bottom-Up ApproachBest Practices for Data Lake DesignMultiple Data LakesAdvantagesDisadvantagesSummary
Data Storage SolutionsData MartsOperational Data StoresData HubsData ProcessesMaster Data ManagementData Virtualization and Data FederationData CatalogsData MarketplacesSummary

Online Transaction Processing Versus Online Analytical ProcessingOperational and Analytical DataSymmetric Multiprocessing and Massively Parallel ProcessingLambda ArchitectureKappa ArchitecturePolyglot Persistence and Polyglot Data StoresSummary
Relational ModelingKeysEntity–Relationship DiagramsNormalization Rules and FormsTracking ChangesDimensional ModelingFacts, Dimensions, and KeysTracking ChangesDenormalizationCommon Data ModelData VaultThe Kimball and Inmon Data Warehousing MethodologiesInmon’s Top-Down MethodologyKimball’s Bottom-Up MethodologyChoosing a MethodologyHybrid ModelsMethodology MythsSummary
ETL Versus ELTReverse ETLBatch Processing Versus Real-Time ProcessingBatch Processing Pros and ConsReal-Time Processing Pros and ConsData GovernanceSummary
The MDW ArchitecturePros and Cons of the MDW ArchitectureCombining the RDW and Data LakeData LakeRelational Data WarehouseStepping Stones to the MDWEDW AugmentationTemporary Data Lake Plus EDWAll-in-OneCase Study: Wilson & Gunkerk’s Strategic Shift to an MDWChallengeSolutionOutcomeSummary
The Data Fabric ArchitectureData Access PoliciesMetadata CatalogMaster Data ManagementData VirtualizationReal-Time ProcessingAPIsServicesProductsWhy Transition from an MDW to a Data Fabric Architecture?Potential DrawbacksSummary
Delta Lake FeaturesPerformance ImprovementsThe Data Lakehouse ArchitectureWhat If You Skip the Relational Data Warehouse?Relational Serving LayerSummary
A Decentralized Data ArchitectureData Mesh HypeDehghani’s Four Principles of Data MeshPrinciple #1: Domain OwnershipPrinciple #2: Data as a ProductPrinciple #3: Self-Serve Data Infrastructure as a PlatformPrinciple #4: Federated Computational GovernanceThe “Pure” Data MeshData DomainsData Mesh Logical ArchitectureDifferent TopologiesData Mesh Versus Data FabricUse CasesSummary
MythsMyth: Using Data Mesh Is a Silver Bullet That Solves All Data Challenges QuicklyMyth: A Data Mesh Will Replace Your Data Lake and Data WarehouseMyth: Data Warehouse Projects Are All Failing, and a Data Mesh Will Solve That ProblemMyth: Building a Data Mesh Means Decentralizing Absolutely EverythingMyth: You Can Use Data Virtualization to Create a Data MeshConcernsPhilosophical and Conceptual MattersCombining Data in a Decentralized EnvironmentOther Issues of DecentralizationComplexityDuplicationFeasibilityPeopleDomain-Level BarriersOrganizational Assessment: Should You Adopt a Data Mesh?Recommendations for Implementing a Successful Data MeshThe Future of Data MeshZooming Out: Understanding Data Architectures and Their ApplicationsSummary
Team Organization: Roles and ResponsibilitiesRoles for MDW, Data Fabric, or Data LakehouseRoles for Data MeshWhy Projects Fail: Pitfalls and PreventionPitfall: Allowing Executives to Think That BI Is “Easy”Pitfall: Using the Wrong TechnologiesPitfall: Gathering Too Many Business RequirementsPitfall: Gathering Too Few Business RequirementsPitfall: Presenting Reports Without Validating Their Contents FirstPitfall: Hiring an Inexperienced Consulting CompanyPitfall: Hiring a Consulting Company That Outsources Development to Offshore WorkersPitfall: Passing Project Ownership Off to ConsultantsPitfall: Neglecting the Need to Transfer Knowledge Back into the OrganizationPitfall: Slashing the Budget Midway Through the ProjectPitfall: Starting with an End Date and Working BackwardPitfall: Structuring the Data Warehouse to Reflect the Source Data Rather Than the Business’s NeedsPitfall: Presenting End Users with a Solution with Slow Response Times or Other Performance IssuesPitfall: Overdesigning (or Underdesigning) Your Data ArchitecturePitfall: Poor Communication Between IT and the Business DomainsTips for SuccessDon’t Skimp on Your InvestmentInvolve Users, Show Them Results, and Get Them ExcitedAdd Value to New Reports and DashboardsAsk End Users to Build a PrototypeFind a Project Champion/SponsorMake a Project Plan That Aims for 80% EfficiencySummary
Choosing a PlatformOpen Source SolutionsOn-Premises SolutionsCloud Provider SolutionsCloud Service ModelsMajor Cloud ProvidersMulti-Cloud SolutionsSoftware FrameworksHadoopDatabricksSnowflakeSummary

Content preview from Deciphering Data Architectures

Chapter 8. Approaches to Data Modeling

Data modeling is a high-level conceptual technique used to design a database. There are several different types of data modeling, including relational modeling and dimensional modeling. The process involves identifying the data that needs to be stored, then creating a structured representation of that data and the relationships among the data, organized into tables and columns. Think of the tables and columns as the logical representation of the database, where the physical data stored in those tables and columns can be in a relational database product or a data lake.

You can use data modeling with any type of database: relational, dimensional, NoSQL, and so on. I recommend dedicating a lot of time to data modeling to ensure that your database is logical, efficient, and easy to use. This ensures maximum performance and makes it easier to retrieve and analyze data.

Relational Modeling

Relational modeling, developed by Edgar F. Codd in 1970 (as mentioned in Chapter 2), is a detailed modeling technique used to design a database. It involves organizing the data into tables and defining the relationships between the tables. In relational databases and relational data warehouses, each table consists of rows (also called records or tuples) and columns (also called fields or attributes). Each row represents a unique instance of the data, and each column represents a specific piece of information about the data.