book

Building an Event-Driven Data Mesh

by Adam Bellemare

April 2023

Intermediate to advanced

259 pages

7h 40m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is Data Mesh?An Event-Driven Data MeshUsing Data in the Operational PlaneThe Data MonolithThe Difficulties of Communicating Data for Operational ConcernsThe Analytical Plane: Data Warehouses and Data LakesThe Organizational Impact of Schema on ReadBad Data: The Costs of InactionCan We Unify Analytical and Operational Workflows?Rethinking Data with Data MeshCommon Objections to an Event-Driven Data MeshProducers Cannot Model Data for Everyone’s Use CasesMaking Multiple Copies of Data Is BadEventual Consistency Is Too Difficult to ManageSummary
Principle 1: Domain OwnershipDomain-Driven Design in BriefSelecting the Data to Expose from Your DomainPrinciple 2: Data as a ProductData Products Provide Immutable and Time-Stamped DataData Products Are MultimodalAccessing a Data Product Via Push or PullThe Three Data Product Alignment TypesEvent-Driven Data Products as Inputs for Operational SystemsPrinciple 3: Federated GovernanceSpecifying Data Product Language, Framework, and API SupportEstablishing Data Product Life Cycle RequirementsEstablishing Data Handling and Infosec PoliciesIdentifying and Standardizing Cross-Domain PolysemesFormalizing Self-Service Platform RequirementsPrinciple 4: Self-Service PlatformDiscovering Data Products and DependenciesData Product Management ControlsData Product Access ControlsCompute and Storage Resources for Building and Using Data ProductsProviding Self-Service Through SaaSSummary
Events, Messages, and RecordsWhat’s an Event Stream? What Is It Not?Ephemeral Message-PassingQueuingConsuming and Using Event-Driven Data ProductsState Events and Event-Carried State TransferMaterializing EventsAggregating EventsThe Kappa ArchitectureThe Lambda Architecture and Why It Doesn’t Work for Data MeshSupporting the Requirements for Kappa ArchitectureSelecting an Event BrokerSummary
Forming a Federated Governance TeamImplementing StandardsSupporting Multimodal Data Product TypesSupporting Data Product SchemasSupporting Programming Languages and FrameworksMetadata Standards and RequirementsEnsuring Cross-Domain Data Product Compatibility and InteroperabilityDefining and Using Common EntitiesEvent Stream Keying and PartitioningTime and Time ZonesWhat Does a Governance Meeting Look Like?1. Identifying Existing Problems2. Drafting Proposals3. Reviewing Proposals4. Implementing Proposals5. Archiving ProposalsData Security and Access PoliciesDisable Data Product Access by DefaultConsider End-to-End EncryptionField-Level EncryptionData Privacy, the Right to Be Forgotten, and Crypto-ShreddingData Product LineageTopology-Based LineageRecord-Based LineageSummary
The Self-Service Platform Maturity ModelLevel 1: The Minimal Viable PlatformThe Schema RegistryAn Extremely Basic Metadata CatalogConnectorsLevel 1 Wrap-Up: How Does It Work?Level 2: The Expanded PlatformFull-Featured Metadata CatalogThe Data Product Management Service and UIService and User IdentitiesBasic Access ControlsStream Processing for Building Data ProductsLevel 2 Wrap-Up: How Does It Work?Level 3: The Mature PlatformAuthentication, Identification, and Access ManagementIntegration with Existing Application Delivery ProcessesProgrammatic Data Product Management APIMonitoring and AlertingMultiregion and Multicloud Data ProductsLevel 3 Wrap-Up: How Does It Work?Summary
A Brief Introduction to Serialization and DeserializationWhat Is a Schema?What Are Our Schema Technology Options?Google’s Protocol Buffers, aka ProtobufApache AvroJSON SchemaSchema Evolution: Changing Your Schemas Through TimeNegotiating a Breaking Schema ChangeStep 1: Design the New Data ModelStep 2: Iterate with Your Existing Consumers and the Federated Governance TeamStep 3. Create a Release Schedule, a Data Migration Plan, and a Deprecation PlanStep 4. Execute the ReleaseThe Role of the Schema RegistryBest Practices for Managing Schemas in Your CodebaseChoosing a Schema TechnologySummary
Introduction to Event TypesExpanding on State Events and Event-Carried State TransferCurrent State EventsBefore/After State EventsDelta EventsEvent Sourcing with Delta EventsWhy Delta Events Don’t Work for Event-Driven Data ProductsMeasurement EventsMeasurement Events Often Form Aggregate-Aligned Data ProductsMeasurement Event Sources May Be LossyMeasurement Events May Power Time-Sensitive ApplicationsHybrid Events—State with a Bit of DeltaNotification EventsSummary
Getting Started: Bootstrapping with ConnectorsDual WritesPolling the Database to Create Data ProductsChange-Data CaptureChange-Data Capture Using a Transactional OutboxDenormalization and EventificationEventification at the Transactional OutboxEventification in a Dedicated ServiceWhat Should Go In the Event? And What Should Stay Out?Slowly Changing DimensionsBootstrapping Cloud Storage Files to an Event StreamSummary
Analytics and the Medallion ArchitectureConnecting Event Streams Into Existing Batch-Data FlowsThrough the Lens of Data Mesh: What’s Going On?Through the Lens of Data Mesh: How Do We Solve It?Balancing File Sizes, SLAs, and LatencyBudget Blues: A Tale of OverspendingExtending the Self-Service Platform for Nonstreaming Data ProductsSummary

Converging on Consistency, One Event at a TimeStrategies for Dealing with Eventual ConsistencyPrevent Failures to Avoid InconsistencyUse Event-Driven Data Products Instead of Request-Response Server API CallsExpose Eventual Consistency in the Server ResponsePlan for New Services and Reprocessing of DataSynchronize Data Products on Time BoundariesOut-of-Order EventsResolving Late-Arriving EventsSummary
Event Streams for Data MeshIntegrating with Existing SystemsOperations, Analytics, and Everything in BetweenSummary

Content preview from Building an Event-Driven Data Mesh

About the Author

Adam Bellemare is a staff technologist, office of the CTO at Confluent. Previously, he was a staff engineer, data platform at Shopify, and he was at Flipp from 2014, first as a senior developer, followed by a staff role. He has also held positions in embedded software development and quality assurance. His expertise includes DevOps (Kafka, Spark, Mesos, Zookeeper Clusters; programmatic building, scaling, destroying); technical leadership (bringing Avro formatting to our data end-to-end, championing Kafka as the event-driven microservice bus, prototyping JRuby, Scala, and Java Kafka clients and focusing on removing technical impediments to allow for product delivery); software development (building microservices in Java and Scala using Spark and Kafka libraries); and data engineering (reshaping the way that behavioral data is collected from user devices and shared with machine learning, billing, and analytics teams). He is the author of Building Event-Driven Microservices (O’Reilly).

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Publisher Resources

ISBN: 9781098127596Errata Page

Building an Event-Driven Data Mesh

by Adam Bellemare

About the Author

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

You might also like

Implementing Data Mesh

Deciphering Data Architectures

Building AI Agents with LangGraph: Creating Agentic Applications with Large Language Models and LangGraph

Data Pipelines with Apache Airflow

Publisher Resources

About the Author

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,and much more.

You might also like

Implementing Data Mesh

Deciphering Data Architectures

Building AI Agents with LangGraph: Creating Agentic Applications with Large Language Models and LangGraph

Data Pipelines with Apache Airflow

Publisher Resources

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.