Streaming Data Mesh

Book description

Data lakes and warehouses have become increasingly fragile, costly, and difficult to maintain as data gets bigger and moves faster. Data meshes can help your organization decentralize data, giving ownership back to the engineers who produced it. This book provides a concise yet comprehensive overview of data mesh patterns for streaming and real-time data services.

Authors Hubert Dulay and Stephen Mooney examine the vast differences between streaming and batch data meshes. Data engineers, architects, data product owners, and those in DevOps and MLOps roles will learn steps for implementing a streaming data mesh, from defining a data domain to building a good data product. Through the course of the book, you'll create a complete self-service data platform and devise a data governance system that enables your mesh to work seamlessly.

With this book, you will:

  • Design a streaming data mesh using Kafka
  • Learn how to identify a domain
  • Build your first data product using self-service tools
  • Apply data governance to the data products you create
  • Learn the differences between synchronous and asynchronous data services
  • Implement self-services that support decentralized data

Table of contents

  1. Preface
    1. Who Should Read This Book
    2. Why We Wrote This Book
    3. Navigating This Book
    4. Conventions Used in This Book
    5. Using Code Examples
    6. O’Reilly Online Learning
    7. How to Contact Us
    8. Acknowledgments
      1. Hubert
      2. Stephen
  2. 1. Data Mesh Introduction
    1. Data Divide
    2. Data Mesh Pillars
      1. Data Ownership
      2. Data as a Product
      3. Federated Computational Data Governance
      4. Self-Service Data Platform
      5. Data Mesh Diagram
    3. Other Similar Architectural Patterns
      1. Data Fabric
      2. Data Gateways and Data Services
      3. Data Democratization
      4. Data Virtualization
    4. Focusing on Implementation
      1. Apache Kafka
      2. AsyncAPI
  3. 2. Streaming Data Mesh Introduction
    1. The Streaming Advantage
      1. Streaming Enables Real-Time Use Cases
      2. Streaming Enables Data Optimization Advantages
      3. Reverse ETL
    2. The Kappa Architecture
      1. Lambda Architecture Introduction
      2. Kappa Architecture Introduction
    3. Summary
  4. 3. Domain Ownership
    1. Identifying Domains
      1. Discernible Domains
      2. Geographic Regions
      3. Hybrid Architecture
      4. Multicloud
    2. Avoiding Ambiguous Domains
    3. Domain-Driven Design
      1. Domain Model
      2. Domain Logic
      3. Bounded Context
      4. The Ubiquitous Language
    4. Data Mesh Domain Roles
      1. Data Product Engineer
      2. Data Product Owner or Data Steward
    5. Streaming Data Mesh Tools and Platforms to Consider
    6. Domain Charge-Backs
    7. Summary
  5. 4. Streaming Data Products
    1. Defining Data Product Requirements
    2. Identifying Data Product Derivatives
      1. Derivatives from Other Domains
    3. Ingesting Data Product Derivatives with Kafka Connect
      1. Consumability
      2. Synchronous Data Sources
      3. Asynchronous Data Sources and Change Data Capture
      4. Debezium Connectors
    4. Transforming Data Derivatives to Data Products
      1. Data Standardization
      2. Protecting Sensitive Information
      3. SQL
      4. Extract, Transform, and Load
    5. Publishing Data Products with AsyncAPI
      1. Registering the Streaming Data Product
      2. Building an AsyncAPI YAML Document
      3. Assigning Data Tags
      4. Versioning
      5. Monitoring
    6. Summary
  6. 5. Federated Computational Data Governance
    1. Data Governance in a Streaming Data Mesh
      1. Data Lineage Graph
      2. Streaming Data Catalog to Organize Data Products
    2. Metadata
      1. Schemas
      2. Lineage
      3. Security
      4. Scalability
    3. Generating the Data Product Page from AsyncAPI
      1. Apicurio Registry
      2. Access Workflow
    4. Centralized Versus Decentralized
      1. Centralized Engineers
      2. Decentralized (Domain) Engineers
    5. Summary
  7. 6. Self-Service Data Infrastructure
    1. Streaming Data Mesh CLI
    2. Resource-Related Commands
      1. Cluster-Related Commands
      2. Topic-Related Commands
      3. The domain Commands
      4. The connect Commands
      5. The streaming Commands
      6. Publishing a Streaming Data Product
    3. Data Governance-Related Services
      1. Security Services
      2. Standards Services
      3. Lineage Services
    4. SaaS Services and APIs
    5. Summary
  8. 7. Architecting a Streaming Data Mesh
    1. Infrastructure
    2. Two Architecture Solutions
      1. Dedicated Infrastructure
      2. Multitenant Infrastructure
    3. Streaming Data Mesh Central Architecture
      1. The Domain Agent (aka Sidecar)
      2. Data Plane
      3. Control Plane
    4. Summary
  9. 8. Building a Decentralized Data Team
    1. The Traditional Data Warehouse Structure
    2. Introducing the Decentralized Team Structure
      1. Empowering People
      2. Working Processes
      3. Fostering Collaboration
      4. Data-Driven Automation
    3. New Roles in Data Domains
      1. New Roles in the Data Plane
      2. New Roles in Data Science and Business Intelligence
  10. 9. Feature Stores
    1. Separating Data Engineering from Data Science
    2. Online and Offline Data Stores
    3. Apache Feast Introduction
    4. Summary
  11. 10. Streaming Data Mesh in Practice
    1. Streaming Data Mesh Example
    2. Deploying an On-Premises Streaming Data Mesh
      1. Installing a Connector
      2. Deploying Clickstream Connector and Auto-Creating Tables
      3. Deploying the Debezium Postgres CDC Connector
      4. Enrichment of Streaming Data
      5. Publishing the Data Product
    3. Consuming Streaming Data Products
    4. Fully Managed SaaS Services
    5. Summary and Considerations
  12. Index
  13. About the Authors

Product information

  • Title: Streaming Data Mesh
  • Author(s): Hubert Dulay, Stephen Mooney
  • Release date: May 2023
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781098130725