Implementing Data Mesh

Book description

As data continues to grow and become more complex, organizations seek innovative solutions to manage their data effectively. Data Mesh is one solution that provides a new approach to managing data in complex organizations. This practical guide offers step-by-step guidance on how to implement data mesh in your organization.

In this book, Jean-Georges Perrin and Eric Broda focus on the key components of data mesh and provide practical advice supported by code. You'll explore a simple and intuitive process for identifying key data mesh components and data products, and learn about a consistent set of interfaces and access methods that make data products easy to consume.

This approach ensures that your data products are easily accessible and the data mesh ecosystem is easy to navigate. With this book, you'll learn how to:

  • Identify, define, and build data products that interoperate within an enterprise data mesh
  • Build a data mesh fabric that binds data products together
  • Build and deploy data products in a data mesh
  • Establish the organizational structure to operate data products, data platforms, and data fabric
  • Learn an innovative architecture that brings data products and data fabric together into the data mesh

About the authors:

Jean-Georges "JG" Perrin is a technology leader focusing on building innovative and modern data platforms.

Eric Broda is a technology executive, practitioner, and founder of a boutique consulting firm that helps global enterprises realize value from data.

Publisher resources

View/Submit Errata

Table of contents

  1. Brief Table of Contents (Not Yet Final)
  2. I. The basics
  3. 1. Understanding Data Mesh - The Essentials
    1. Adopting Local Autonomy, Speed, and Agility
    2. Today’s Data Challenge
    3. Turning Principles into Practice
  4. 2. Applying Data Mesh Principles
    1. What is a Data Mesh?
    2. Data Mesh Principles
    3. Defining a “Good” Data Product
    4. Defining a Principled Data Product
    5. Defining a FAIR Data Product
    6. Defining an Enterprise Grade Data Product
    7. Defining a Valuable Data Product
    8. Defining a Balanced Data Product
    9. A “Good” Data Product is More than Just Data
    10. Defining a Valuable Data Product
    11. A “Good” Data Product has an Empowered Data Product
    12. Conclusion
  5. 3. Our Case Study - Climate Quantum Inc.
    1. Making Climate Data Easier to Find, Consume, Share, and Trust
    2. Introducing Climate Quantum Inc
    3. Applying Climate Quantum Inc to Your Enterprise
  6. II. Designing, building, and deploying Data Mesh
  7. 4. Defining the Data Mesh Architecture
    1. Definition
    2. Data Product Harness
    3. Run-Time
    4. Ingestion Interfaces
    5. Consumption Interfaces
    6. Policy Enforcement
    7. Operations Experience
    8. Discoverability Interfaces
    9. Observability Interfaces
    10. Governance Interfaces
    11. Control Interfaces
    12. Data Product Artifacts
    13. Data Contracts
    14. Data Product Governance
    15. Data Mesh Architecture
      1. Data Mesh Marketplace
      2. Data Mesh Registry
      3. Data Mesh Console
      4. Data Mesh Fabric
    16. Data Product Actors
    17. Climate Quantum Use Case
  8. 5. Driving Data Products with Data Contracts
    1. Bringing Value Through Trust
    2. Navigating the data contract
      1. Going through the theory
      2. Stacking up good information
      3. It’s all about proper versioning
      4. Keeping it simple and semantic
      5. Walking through an example: complementing tribal knowledge
    3. What is Data QoS, and why is it critical?
      1. Data Quality of Service (Data QoS)
      2. Representation
    4. Why does it matter?
    5. Data Quality is not enough
      1. Accuracy (Ac)
      2. Completeness (Cp)
      3. Conformity (Cf)
      4. Consistency (Cs)
      5. Coverage (Cv)
      6. Timeliness (Tm)
      7. Uniqueness (Uq)
    6. Service-levels complement quality
      1. Availability (Av)
      2. Throughput (Th)
      3. Error rate (Er)
      4. General availability (Ga)
      5. End of support (Es)
      6. End of life (El)
      7. Retention (Re)
      8. Frequency of update (Fy)
      9. Latency (Ly)
      10. Time to detect (an issue) (Td)
      11. Time to notify (Tn)
      12. Time to repair (Tr)
    7. Applying Data QoS to the data contract
      1. Checking conformity of measurements
      2. Completeness
      3. Accuracy
      4. Engaging service levels
    8. Summary
  9. 6. Data Mesh and Generative AI
    1. Large Language Models (LLMs)
      1. Embeddings
      2. Vector Databases
      3. Challenges
    2. Data Mesh and Generative AI
      1. An Architecture for Generative-AI
    3. Enterprise Data Mesh and Generative AI
      1. Applying Generative-AI to Climate Data Inc
    4. Summary
  10. III. Teams, operating models, and roadmaps for Data Mesh
  11. 7. Establishing the Data Mesh Team
    1. Team Topologies in Data Mesh
    2. The Data Product Team
    3. Key Roles and Responsibilities
      1. Data Product Owner
      2. Release Manager
      3. Metadata and Governance Manager
      4. Data and Security Manager
      5. Consumption Services Manager
      6. Ingestion Services Manager
    4. Data Product Skills Matrix
    5. Benefits
    6. Challenges
    7. Summary
  12. 8. Defining a Data Mesh Operating Model
    1. Introduction
    2. Characteristics of an Operating Model
    3. Data Mesh Ecosystem Operating Model
    4. Data Certification vs Traditional Data Governance
    5. Impact of Operating Model Choices
    6. Data Mesh as Loosely Coupled Regional Ecosystems

Product information

  • Title: Implementing Data Mesh
  • Author(s): Jean-Georges Perrin, Eric Broda
  • Release date: September 2024
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781098156220