book

Implementing Data Mesh

by Jean-Georges Perrin, Eric Broda

September 2024

Intermediate to advanced

268 pages

7h 13m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Includes

Includes Quizzes

Who Is This Book For? Overview of the Parts and ChaptersWhat This Book Isn’t Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
Making Data AgileLocal Autonomy + Speed = AgilitySolving Today’s Data ChallengesBridging Data SilosShifting Toward Higher Quality DataTransforming Data GovernanceData Volume, Variety, and VariabilityTurning Principles into PracticeSummary
Data Mesh PrinciplesData as a ProductDecentralized Domain OwnershipSelf-Serve Data PlatformFederated Computational GovernanceDefining a “Good” Data ProductDefining a Principled Data ProductDefining a FAIR Data ProductDefining an Enterprise-Grade Data ProductDefining a Valuable Data ProductDefining a Balanced Data ProductDefining a Modern Data Product—More than Just “Regular” DataDefining a Practical Data Product LifecycleDefining a Practical Data Mesh RoadmapA “Good” Data Product Has an Empowered Data Product OwnerIdentifying a Data ProductSummary
Making Climate Data Easier to Find, Consume, Share, and TrustIntroducing Climate Quantum Inc.Climate Quantum Data LandscapeApplying Climate Quantum Inc. to Your EnterpriseSummary
Data Product ArchitectureData Product ArtifactsDevelopment Architecture ComponentsRuntime Architecture ComponentsOperations Architecture ComponentsData Mesh ArchitectureData Mesh Marketplace and RegistryData Mesh Backbone ServicesClimate Quantum Use Case ConsiderationsSummary
Bringing Value Through TrustNavigating the Data ContractGoing Through the TheoryStacking Up Good InformationIt’s All About Proper VersioningKeeping It Simple and SemanticWalking Through an Example: Complementing Tribal KnowledgeWhat is Data QoS and Why Is It Critical?RepresentationWhy Does It Matter?Why Data Quality Is Not EnoughService Levels Complement QualityApplying Data QoS to the Data ContractChecking Conformity of MeasurementsCompletenessAccuracyEngaging Service LevelsSummary
Anatomy of a Data ProductThe Data Contract to the RescueConnecting Your Data SourcesEnsuring Higher Data Quality with ObservabilityGetting Faster Data DiscoveryEnabling Operability and ControlBuilding and PackagingSummary

The Three Planes The Infrastructure Plane Remains KeyThe Data Product Experience Plane Is IndependentThe Mesh Experience Plane Is About SynergyBuilding a Capability Model for Each PlaneCapabilities of the Infrastructure Experience PlaneCapabilities of the Data Product Experience PlaneCapabilities of the Mesh Experience PlaneGathering Key Metrics Through Feedback LoopsSummary
Registering Your Data ProductConnecting to Your Products in the MeshMeshing Data Products TogetherDescribing Your Data LineageNotification Through the MeshSummary
Making Data Products Discoverable, Observable, and SecureData Product Discovery InterfaceData Product Observability InterfaceData Product Control InterfaceData Product SecurityClimate Quantum Use Case ConsiderationsToward Dynamic Data ProductsThe Power of Dynamic Data ProductsSummary
Challenges with Traditional Data CatalogsThe Data Mesh MarketplaceA Window into Data MeshThe Consumer User JourneyThe Producer User JourneySelf-Publishing of Data ProductsThe Power of DiscoverabilityBootstrapping Your MarketplaceClimate Quantum’s Data Mesh MarketplaceSummary
Data GovernanceData Product CertificationFederated Data Product CertificationImplementing Data Product Certification Data Contracts and CertificationClimate Quantum’s Certification ApproachSummary
The Modern Manufacturing Supply ChainThe Modern Software FactoryThe Data Product FactoryThe Data Product Supply ChainClimate Quantum’s Data Product Supply ChainSummary
Generative AI BackgroundLarge Language Models EmbeddingsVector DatabasesPromptsChallengesData Mesh and Generative AIImprove Data Quality with Generative AIElevate Models as Primary ArtifactsConsume Models Inside Data ProductsGenerate Data Product CodeComplement Data Product CapabilitiesClimate Quantum and Generative AISimplifying Climate Data SearchAnalyzing and Summarizing Climate Data Creating Disclosure ReportsGenerating Code to Simplify Climate Data ProcessingSummary
Team Topologies in Data MeshThe Data Product TeamPlatform TeamsEnabling TeamsClimate Quantum TeamsSummary
Characteristics of an Operating ModelThe Operating Model ContinuumCentralized OrganizationsMatrixed OrganizationsFederated OrganizationsDistributed OrganizationsData Mesh Operating ModelData Mesh Maturity ModelClimate Quantum’s Data Mesh Operating ModelSummary
Roadmap StructureStrategy and Roadmap StreamTechnology StreamFactory StreamOperating Model StreamSocialization StreamRollout StreamClimate Quantum’s Data Mesh RoadmapSummary

Content preview from Implementing Data Mesh

Foreword

When I first encountered the Data Mesh concept in late 2020, I was much more steeped in the best practices of the operational world, especially site reliability engineering (SRE) and distributed systems. Zhamak Dehghani, the creator of the Data Mesh concept, proposed a number of ways of doing analytics, machine learning, and data work that felt very familiar and a bit obvious because of how long they had been adopted in the software world, like shifting ownership left, product thinking, continuous integration and continuous delivery (CI/CD), etc.

However, after I spent a few weeks digging deeper, including into what people were actually saying about Data Mesh in posts on LinkedIn and Twitter—so not just the articles, presentations, and podcasts—there was a resounding question underlying everything: OK, but how? The data world in general hadn’t done work in any way close to what Data Mesh calls for, and trying to simply jump into entirely new ways is 1) disruptive in general and 2) abhorrent to most data people....

Because of all the questions around Data Mesh, I created a community that is now over 10,000 people strong (Data Mesh Learning) and a podcast as well to explore this with more than 300 episodes (Data Mesh Radio). Because that question is still the most pertinent and pernicious one about Data Mesh today: OK, but how?

From the very early days of the community, both Jean-Georges and Eric have been helping people to explore this question. Not just what is Data Mesh—I ...