book

Implementing Data Mesh

by Jean-Georges Perrin, Eric Broda

September 2024

Intermediate to advanced

268 pages

7h 13m

English

O'Reilly Media, Inc.

Audiobook available

Read now

Unlock full access

Includes

Quizzes

Foreword
Preface
Who Is This Book For? Overview of the Parts and ChaptersWhat This Book Isn’t Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
I. The Basics
1. Understanding Data Mesh: The Essentials
Making Data AgileLocal Autonomy + Speed = AgilitySolving Today’s Data ChallengesBridging Data SilosShifting Toward Higher Quality DataTransforming Data GovernanceData Volume, Variety, and VariabilityTurning Principles into PracticeSummary
2. Applying Data Mesh Principles
Data Mesh PrinciplesData as a ProductDecentralized Domain OwnershipSelf-Serve Data PlatformFederated Computational GovernanceDefining a “Good” Data ProductDefining a Principled Data ProductDefining a FAIR Data ProductDefining an Enterprise-Grade Data ProductDefining a Valuable Data ProductDefining a Balanced Data ProductDefining a Modern Data Product—More than Just “Regular” DataDefining a Practical Data Product LifecycleDefining a Practical Data Mesh RoadmapA “Good” Data Product Has an Empowered Data Product OwnerIdentifying a Data ProductSummary
3. Our Case Study: Climate Quantum Inc.
Making Climate Data Easier to Find, Consume, Share, and TrustIntroducing Climate Quantum Inc.Climate Quantum Data LandscapeApplying Climate Quantum Inc. to Your EnterpriseSummary
II. Designing, Building, and Deploying Data Mesh
4. Defining the Data Mesh Architecture
Data Product ArchitectureData Product ArtifactsDevelopment Architecture ComponentsRuntime Architecture ComponentsOperations Architecture ComponentsData Mesh ArchitectureData Mesh Marketplace and RegistryData Mesh Backbone ServicesClimate Quantum Use Case ConsiderationsSummary
5. Driving Data Products with Data Contracts
Bringing Value Through TrustNavigating the Data ContractGoing Through the TheoryStacking Up Good InformationIt’s All About Proper VersioningKeeping It Simple and SemanticWalking Through an Example: Complementing Tribal KnowledgeWhat is Data QoS and Why Is It Critical?RepresentationWhy Does It Matter?Why Data Quality Is Not EnoughService Levels Complement QualityApplying Data QoS to the Data ContractChecking Conformity of MeasurementsCompletenessAccuracyEngaging Service LevelsSummary
6. Building Your First Data Product
Anatomy of a Data ProductThe Data Contract to the RescueConnecting Your Data SourcesEnsuring Higher Data Quality with ObservabilityGetting Faster Data DiscoveryEnabling Operability and ControlBuilding and PackagingSummary

7. Aligning with the Experience Planes
The Three Planes The Infrastructure Plane Remains KeyThe Data Product Experience Plane Is IndependentThe Mesh Experience Plane Is About SynergyBuilding a Capability Model for Each PlaneCapabilities of the Infrastructure Experience PlaneCapabilities of the Data Product Experience PlaneCapabilities of the Mesh Experience PlaneGathering Key Metrics Through Feedback LoopsSummary
8. Meshing Your Data Products
Registering Your Data ProductConnecting to Your Products in the MeshMeshing Data Products TogetherDescribing Your Data LineageNotification Through the MeshSummary
III. GenAI, Teams, Operating Model, and Roadmap for Data Mesh
9. Running and Operating Your Data Mesh
Making Data Products Discoverable, Observable, and SecureData Product Discovery InterfaceData Product Observability InterfaceData Product Control InterfaceData Product SecurityClimate Quantum Use Case ConsiderationsToward Dynamic Data ProductsThe Power of Dynamic Data ProductsSummary
10. Creating a Data Mesh Marketplace
Challenges with Traditional Data CatalogsThe Data Mesh MarketplaceA Window into Data MeshThe Consumer User JourneyThe Producer User JourneySelf-Publishing of Data ProductsThe Power of DiscoverabilityBootstrapping Your MarketplaceClimate Quantum’s Data Mesh MarketplaceSummary
11. Establishing Data Mesh Governance
Data GovernanceData Product CertificationFederated Data Product CertificationImplementing Data Product Certification Data Contracts and CertificationClimate Quantum’s Certification ApproachSummary
12. Understanding Data Product Supply Chains
The Modern Manufacturing Supply ChainThe Modern Software FactoryThe Data Product FactoryThe Data Product Supply ChainClimate Quantum’s Data Product Supply ChainSummary
13. Integrating Data Mesh and Generative AI
Generative AI BackgroundLarge Language Models EmbeddingsVector DatabasesPromptsChallengesData Mesh and Generative AIImprove Data Quality with Generative AIElevate Models as Primary ArtifactsConsume Models Inside Data ProductsGenerate Data Product CodeComplement Data Product CapabilitiesClimate Quantum and Generative AISimplifying Climate Data SearchAnalyzing and Summarizing Climate Data Creating Disclosure ReportsGenerating Code to Simplify Climate Data ProcessingSummary
14. Establishing Data Mesh Teams
Team Topologies in Data MeshThe Data Product TeamPlatform TeamsEnabling TeamsClimate Quantum TeamsSummary
15. Defining a Data Mesh Operating Model
Characteristics of an Operating ModelThe Operating Model ContinuumCentralized OrganizationsMatrixed OrganizationsFederated OrganizationsDistributed OrganizationsData Mesh Operating ModelData Mesh Maturity ModelClimate Quantum’s Data Mesh Operating ModelSummary
16. Establishing a Practical Data Mesh Roadmap
Roadmap StructureStrategy and Roadmap StreamTechnology StreamFactory StreamOperating Model StreamSocialization StreamRollout StreamClimate Quantum’s Data Mesh RoadmapSummary
Index
About the Authors

Content preview from Implementing Data Mesh

About the Authors

Jean-Georges “JGP” Perrin is the chief innovation officer at AbeaData, focusing on building innovative and modern data tooling. He is also chair of the Open Data Contract Standard (ODCS) at the Linux Foundation project Bitol, cofounder of AIDA User Group, and author of multiple books, including Implementing Data Mesh (O’Reilly) and Spark in Action, 2nd edition (Manning). He is passionate about software engineering and all things data. His latest endeavors bring him to more and more data engineering, data governance, industrialization of data science, and his favorite theme, Data Mesh. He is proud to have been recognized as a Lifetime IBM Champion, a PayPal Champion, and a Data Mesh MVP. Jean-Georges shares over 25 years of experience in the IT industry as a presenter and participant at conferences and publishing articles in print and online media. His blog is visible at http://jgp.ai. He enjoys exploring upstate New York and New England with his wife and kids when not immersed in tech, which he loves.

Eric Broda has been working in financial services for over three decades and has been a technology executive at, and consultant to, large banks, insurers, and payments firms. Today, Eric runs a boutique consulting company that designs, architects, and implements Data Mesh, data ecosystem, and generative AI solutions. Eric is also playing an active role in addressing the global climate change challenge where he is leading a team at an international nonprofit that ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098156213Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Implementing Data Mesh

by Jean-Georges Perrin, Eric Broda

About the Authors

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.