book

Data Mesh

by Zhamak Dehghani

March 2022

Beginner to intermediate

384 pages

10h 54m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Why I Wrote This Book and Why NowWho Should Read This BookHow to Read This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
Data Mesh in ActionA Culture of Data Curiosity and ExperimentationAn Embedded Partnership with Data and MLThe Invisible Platform and PoliciesLimitless Scale with Autonomous Data ProductsThe Positive Network EffectWhy Transform to Data Mesh?The Way Forward
The OutcomesThe ShiftsThe PrinciplesPrinciple of Domain OwnershipPrinciple of Data as a ProductPrinciple of the Self-Serve Data PlatformPrinciple of Federated Computational GovernanceInterplay of the PrinciplesData Mesh Model at a GlanceThe DataOperational DataAnalytical DataThe Origin
A Brief Background on Domain-Driven DesignApplying DDD’s Strategic Design to DataDomain Data ArchetypesSource-Aligned Domain DataAggregate Domain DataConsumer-Aligned Domain DataTransition to Domain OwnershipPush Data Ownership UpstreamDefine Multiple Connected ModelsEmbrace the Most Relevant Domain Data: Don’t Expect a Single Source of TruthHide the Data Pipelines as Domains’ Internal ImplementationRecap
Applying Product Thinking to DataBaseline Usability Attributes of a Data ProductTransition to Data as a ProductInclude Data Product Ownership in DomainsReframe the Nomenclature to Create ChangeThink of Data as a Product, Not a Mere AssetEstablish a Trust-But-Verify Data CultureJoin Data and Compute as One Logical UnitRecap
Data Mesh Platform: Compare and ContrastServing Autonomous Domain-Oriented TeamsManaging Autonomous and Interoperable Data ProductsA Continuous Platform of Operational and Analytical CapabilitiesDesigned for a Generalist MajorityFavoring Decentralized TechnologiesDomain AgnosticData Mesh Platform ThinkingEnable Autonomous Teams to Get Value from DataExchange Value with Autonomous and Interoperable Data ProductsAccelerate Exchange of Value by Lowering the Cognitive LoadScale Out Data SharingSupport a Culture of Embedded InnovationTransition to a Self-Serve Data Mesh PlatformDesign the APIs and Protocols FirstPrepare for Generalist AdoptionDo an Inventory and SimplifyCreate Higher-Level APIs to Manage Data ProductsBuild Experiences, Not MechanismsBegin with the Simplest Foundation, Then Harvest to EvolveRecap
Apply Systems Thinking to Data Mesh GovernanceMaintain Dynamic Equilibrium Between Domain Autonomy and Global InteroperabilityEmbrace Dynamic Topology as a Default StateUtilize Automation and the Distributed ArchitectureApply Federation to the Governance ModelFederated TeamGuiding ValuesPoliciesIncentivesApply Computation to the Governance ModelStandards as CodePolicies as CodeAutomated TestsAutomated MonitoringTransition to Federated Computational GovernanceDelegate Accountability to DomainsEmbed Policy Execution in Each Data ProductAutomate Enablement and Monitoring over InterventionsModel the GapsMeasure the Network EffectEmbrace Change over ConstancyRecap

Great Expectations of DataThe Great Divide of DataScale: Encounter of a New KindBeyond OrderApproaching the Plateau of ReturnRecap
Respond Gracefully to Change in a Complex BusinessAlign Business, Tech, and Now Analytical DataClose the Gap Between Analytical and Operational DataLocalize Data Changes to Business DomainsReduce Accidental Complexity of Pipelines and Copying DataSustain Agility in the Face of GrowthRemove Centralized and Monolithic BottlenecksReduce Coordination of Data PipelinesReduce Coordination of Data GovernanceEnable AutonomyIncrease the Ratio of Value from Data to InvestmentAbstract Technical Complexity with a Data PlatformEmbed Product Thinking EverywhereGo Beyond the BoundariesRecap
Evolution of Analytical Data ArchitecturesFirst Generation: Data Warehouse ArchitectureSecond Generation: Data Lake ArchitectureThird Generation: Multimodal Cloud ArchitectureCharacteristics of Analytical Data ArchitectureMonolithicCentralized Data OwnershipTechnology OrientedRecap
Domain-Oriented Analytical Data Sharing InterfacesOperational Interface DesignAnalytical Data Interface DesignInterdomain Analytical Data DependenciesData Product as an Architecture QuantumA Data Product’s Structural ComponentsData Product Data Sharing InteractionsData Discovery and Observability APIsThe Multiplane Data PlatformA Platform PlaneData Infrastructure (Utility) PlaneData Product Experience PlaneMesh Experience PlaneExampleEmbedded Computational PoliciesData Product SidecarData Product Computational ContainerControl PortRecap
Design a Platform Driven by User JourneysData Product Developer JourneyIncept, Explore, Bootstrap, and SourceBuild, Test, Deploy, and RunMaintain, Evolve, and RetireData Product Consumer JourneyIncept, Explore, Bootstrap, SourceBuild, Test, Deploy, RunMaintain, Evolve, and RetireRecap
Data Product AffordancesData Product Architecture CharacteristicsDesign Influenced by the Simplicity of Complex Adaptive SystemsEmergent Behavior from Simple Local RulesNo Central OrchestratorRecap
Serve DataThe Needs of Data UsersServe Data Design PropertiesServe Data DesignConsume DataArchetypes of Data SourcesLocality of Data ConsumptionData Consumption DesignTransform DataProgrammatic Versus Nonprogrammatic TransformationDataflow-Based TransformationML as TransformationTime-Variant TransformationTransformation DesignRecap
Discover, Understand, Trust, and ExploreBegin Discovery with Self-RegistrationDiscover the Global URIUnderstand Semantic and Syntax ModelsEstablish Trust with Data GuaranteesExplore the Shape of DataLearn with DocumentationDiscover, Explore, and Understand DesignCompose DataConsume Data Design PropertiesTraditional Approaches to Data ComposabilityCompose Data DesignRecap
Manage the Life CycleManage Life-Cycle DesignData Product Manifest ComponentsGovern DataGovern Data DesignStandardize PoliciesData and Policy IntegrationLinking PoliciesObserve, Debug, and AuditObservability DesignRecap
Should You Adopt Data Mesh Today?Data Mesh as an Element of Data StrategyData Mesh Execution FrameworkBusiness-Driven ExecutionEnd-to-End and Iterative ExecutionEvolutionary ExecutionRecap
ChangeCultureValuesRewardIntrinsic MotivationsExtrinsic MotivationsStructureOrganization Structure AssumptionsDiscover Data Product BoundariesPeopleRolesSkillset DevelopmentProcessKey Process ChangesRecap

Content preview from Data Mesh

Part V. How to Get Started

A journey of a thousand miles begins with a single step.

Lao Tzu

We have both made it to the last part of this book. Well done!

Are you considering adopting data mesh for your organization or helping others to do so? Are you in a position to influence, lead, or manage the execution of data mesh? Do you need some help on how to approach such a transformation organizationally and where to get started?

If you answered yes to these questions, this part is for you.

I assume that you have a solid understanding of data mesh foundational principles from Part I and understand the motivation and objectives of data mesh from Part II.

Scope

Data mesh is an element of a data strategy that fosters a data-driven organization to get value from data at scale.

Execution of this strategy has multiple facets. It affects the teams, their accountability structure, and delineation of responsibilities between domains and platform teams. It influences the culture, how organizations value and measure their data-oriented success. It changes the operating model and how local and global decisions around data availability, security, and interoperability are made. It introduces a new architecture that supports a decentralized model of data sharing.

In this part of the book, I describe a high-level approach to start and guide the execution of this multifaceted change. In Chapter 15, “Strategy and Execution”, I introduce an evolutionary approach to adopt data mesh incrementally, ...