book

Building an Event-Driven Data Mesh

by Adam Bellemare

April 2023

Intermediate to advanced

259 pages

7h 40m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is Data Mesh?An Event-Driven Data MeshUsing Data in the Operational PlaneThe Data MonolithThe Difficulties of Communicating Data for Operational ConcernsThe Analytical Plane: Data Warehouses and Data LakesThe Organizational Impact of Schema on ReadBad Data: The Costs of InactionCan We Unify Analytical and Operational Workflows?Rethinking Data with Data MeshCommon Objections to an Event-Driven Data MeshProducers Cannot Model Data for Everyone’s Use CasesMaking Multiple Copies of Data Is BadEventual Consistency Is Too Difficult to ManageSummary
Principle 1: Domain OwnershipDomain-Driven Design in BriefSelecting the Data to Expose from Your DomainPrinciple 2: Data as a ProductData Products Provide Immutable and Time-Stamped DataData Products Are MultimodalAccessing a Data Product Via Push or PullThe Three Data Product Alignment TypesEvent-Driven Data Products as Inputs for Operational SystemsPrinciple 3: Federated GovernanceSpecifying Data Product Language, Framework, and API SupportEstablishing Data Product Life Cycle RequirementsEstablishing Data Handling and Infosec PoliciesIdentifying and Standardizing Cross-Domain PolysemesFormalizing Self-Service Platform RequirementsPrinciple 4: Self-Service PlatformDiscovering Data Products and DependenciesData Product Management ControlsData Product Access ControlsCompute and Storage Resources for Building and Using Data ProductsProviding Self-Service Through SaaSSummary
Events, Messages, and RecordsWhat’s an Event Stream? What Is It Not?Ephemeral Message-PassingQueuingConsuming and Using Event-Driven Data ProductsState Events and Event-Carried State TransferMaterializing EventsAggregating EventsThe Kappa ArchitectureThe Lambda Architecture and Why It Doesn’t Work for Data MeshSupporting the Requirements for Kappa ArchitectureSelecting an Event BrokerSummary
Forming a Federated Governance TeamImplementing StandardsSupporting Multimodal Data Product TypesSupporting Data Product SchemasSupporting Programming Languages and FrameworksMetadata Standards and RequirementsEnsuring Cross-Domain Data Product Compatibility and InteroperabilityDefining and Using Common EntitiesEvent Stream Keying and PartitioningTime and Time ZonesWhat Does a Governance Meeting Look Like?1. Identifying Existing Problems2. Drafting Proposals3. Reviewing Proposals4. Implementing Proposals5. Archiving ProposalsData Security and Access PoliciesDisable Data Product Access by DefaultConsider End-to-End EncryptionField-Level EncryptionData Privacy, the Right to Be Forgotten, and Crypto-ShreddingData Product LineageTopology-Based LineageRecord-Based LineageSummary
The Self-Service Platform Maturity ModelLevel 1: The Minimal Viable PlatformThe Schema RegistryAn Extremely Basic Metadata CatalogConnectorsLevel 1 Wrap-Up: How Does It Work?Level 2: The Expanded PlatformFull-Featured Metadata CatalogThe Data Product Management Service and UIService and User IdentitiesBasic Access ControlsStream Processing for Building Data ProductsLevel 2 Wrap-Up: How Does It Work?Level 3: The Mature PlatformAuthentication, Identification, and Access ManagementIntegration with Existing Application Delivery ProcessesProgrammatic Data Product Management APIMonitoring and AlertingMultiregion and Multicloud Data ProductsLevel 3 Wrap-Up: How Does It Work?Summary
A Brief Introduction to Serialization and DeserializationWhat Is a Schema?What Are Our Schema Technology Options?Google’s Protocol Buffers, aka ProtobufApache AvroJSON SchemaSchema Evolution: Changing Your Schemas Through TimeNegotiating a Breaking Schema ChangeStep 1: Design the New Data ModelStep 2: Iterate with Your Existing Consumers and the Federated Governance TeamStep 3. Create a Release Schedule, a Data Migration Plan, and a Deprecation PlanStep 4. Execute the ReleaseThe Role of the Schema RegistryBest Practices for Managing Schemas in Your CodebaseChoosing a Schema TechnologySummary
Introduction to Event TypesExpanding on State Events and Event-Carried State TransferCurrent State EventsBefore/After State EventsDelta EventsEvent Sourcing with Delta EventsWhy Delta Events Don’t Work for Event-Driven Data ProductsMeasurement EventsMeasurement Events Often Form Aggregate-Aligned Data ProductsMeasurement Event Sources May Be LossyMeasurement Events May Power Time-Sensitive ApplicationsHybrid Events—State with a Bit of DeltaNotification EventsSummary
Getting Started: Bootstrapping with ConnectorsDual WritesPolling the Database to Create Data ProductsChange-Data CaptureChange-Data Capture Using a Transactional OutboxDenormalization and EventificationEventification at the Transactional OutboxEventification in a Dedicated ServiceWhat Should Go In the Event? And What Should Stay Out?Slowly Changing DimensionsBootstrapping Cloud Storage Files to an Event StreamSummary
Analytics and the Medallion ArchitectureConnecting Event Streams Into Existing Batch-Data FlowsThrough the Lens of Data Mesh: What’s Going On?Through the Lens of Data Mesh: How Do We Solve It?Balancing File Sizes, SLAs, and LatencyBudget Blues: A Tale of OverspendingExtending the Self-Service Platform for Nonstreaming Data ProductsSummary

Converging on Consistency, One Event at a TimeStrategies for Dealing with Eventual ConsistencyPrevent Failures to Avoid InconsistencyUse Event-Driven Data Products Instead of Request-Response Server API CallsExpose Eventual Consistency in the Server ResponsePlan for New Services and Reprocessing of DataSynchronize Data Products on Time BoundariesOut-of-Order EventsResolving Late-Arriving EventsSummary
Event Streams for Data MeshIntegrating with Existing SystemsOperations, Analytics, and Everything in BetweenSummary

Overview

The exponential growth of data combined with the need to derive real-time business value is a critical issue today. An event-driven data mesh can power real-time operational and analytical workloads, all from a single set of data product streams. With practical real-world examples, this book shows you how to successfully design and build an event-driven data mesh.

Building an Event-Driven Data Mesh provides:

Practical tips for iteratively building your own event-driven data mesh, including hurdles you'll experience, possible solutions, and how to obtain real value as soon as possible
Solutions to pitfalls you may encounter when moving your organization from monoliths to event-driven architectures
A clear understanding of how events relate to systems and other events in the same stream and across streams
A realistic look at event modeling options, such as fact, delta, and command type events, including how these choices will impact your data products
Best practices for handling events at scale, privacy, and regulatory compliance
Advice on asynchronous communication and handling eventual consistency

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Implementing Data Mesh

Jean-Georges Perrin, Eric Broda

Deciphering Data Architectures

James Serra

Building AI Agents with LangGraph: Creating Agentic Applications with Large Language Models and LangGraph

Sajal Sharma

Data Pipelines with Apache Airflow

Julian de Ruiter, Bas Harenslak

Publisher Resources

ISBN: 9781098127596Errata Page