book

Building an Event-Driven Data Mesh

Name: Building an Event-Driven Data Mesh
Author: Adam Bellemare
ISBN: 9781098127602

by Adam Bellemare

April 2023

Intermediate to advanced

259 pages

7h 40m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
1. Event-Driven Data Communication
What Is Data Mesh?An Event-Driven Data MeshUsing Data in the Operational PlaneThe Data MonolithThe Difficulties of Communicating Data for Operational ConcernsThe Analytical Plane: Data Warehouses and Data LakesThe Organizational Impact of Schema on ReadBad Data: The Costs of InactionCan We Unify Analytical and Operational Workflows?Rethinking Data with Data MeshCommon Objections to an Event-Driven Data MeshProducers Cannot Model Data for Everyone’s Use CasesMaking Multiple Copies of Data Is BadEventual Consistency Is Too Difficult to ManageSummary
2. Data Mesh
Principle 1: Domain OwnershipDomain-Driven Design in BriefSelecting the Data to Expose from Your DomainPrinciple 2: Data as a ProductData Products Provide Immutable and Time-Stamped DataData Products Are MultimodalAccessing a Data Product Via Push or PullThe Three Data Product Alignment TypesEvent-Driven Data Products as Inputs for Operational SystemsPrinciple 3: Federated GovernanceSpecifying Data Product Language, Framework, and API SupportEstablishing Data Product Life Cycle RequirementsEstablishing Data Handling and Infosec PoliciesIdentifying and Standardizing Cross-Domain PolysemesFormalizing Self-Service Platform RequirementsPrinciple 4: Self-Service PlatformDiscovering Data Products and DependenciesData Product Management ControlsData Product Access ControlsCompute and Storage Resources for Building and Using Data ProductsProviding Self-Service Through SaaSSummary
3. Event Streams for Data Mesh
Events, Messages, and RecordsWhat’s an Event Stream? What Is It Not?Ephemeral Message-PassingQueuingConsuming and Using Event-Driven Data ProductsState Events and Event-Carried State TransferMaterializing EventsAggregating EventsThe Kappa ArchitectureThe Lambda Architecture and Why It Doesn’t Work for Data MeshSupporting the Requirements for Kappa ArchitectureSelecting an Event BrokerSummary
4. Federated Governance
Forming a Federated Governance TeamImplementing StandardsSupporting Multimodal Data Product TypesSupporting Data Product SchemasSupporting Programming Languages and FrameworksMetadata Standards and RequirementsEnsuring Cross-Domain Data Product Compatibility and InteroperabilityDefining and Using Common EntitiesEvent Stream Keying and PartitioningTime and Time ZonesWhat Does a Governance Meeting Look Like?1. Identifying Existing Problems2. Drafting Proposals3. Reviewing Proposals4. Implementing Proposals5. Archiving ProposalsData Security and Access PoliciesDisable Data Product Access by DefaultConsider End-to-End EncryptionField-Level EncryptionData Privacy, the Right to Be Forgotten, and Crypto-ShreddingData Product LineageTopology-Based LineageRecord-Based LineageSummary
5. Self-Service Data Platform
The Self-Service Platform Maturity ModelLevel 1: The Minimal Viable PlatformThe Schema RegistryAn Extremely Basic Metadata CatalogConnectorsLevel 1 Wrap-Up: How Does It Work?Level 2: The Expanded PlatformFull-Featured Metadata CatalogThe Data Product Management Service and UIService and User IdentitiesBasic Access ControlsStream Processing for Building Data ProductsLevel 2 Wrap-Up: How Does It Work?Level 3: The Mature PlatformAuthentication, Identification, and Access ManagementIntegration with Existing Application Delivery ProcessesProgrammatic Data Product Management APIMonitoring and AlertingMultiregion and Multicloud Data ProductsLevel 3 Wrap-Up: How Does It Work?Summary
6. Event Schemas
A Brief Introduction to Serialization and DeserializationWhat Is a Schema?What Are Our Schema Technology Options?Google’s Protocol Buffers, aka ProtobufApache AvroJSON SchemaSchema Evolution: Changing Your Schemas Through TimeNegotiating a Breaking Schema ChangeStep 1: Design the New Data ModelStep 2: Iterate with Your Existing Consumers and the Federated Governance TeamStep 3. Create a Release Schedule, a Data Migration Plan, and a Deprecation PlanStep 4. Execute the ReleaseThe Role of the Schema RegistryBest Practices for Managing Schemas in Your CodebaseChoosing a Schema TechnologySummary
7. Designing Events
Introduction to Event TypesExpanding on State Events and Event-Carried State TransferCurrent State EventsBefore/After State EventsDelta EventsEvent Sourcing with Delta EventsWhy Delta Events Don’t Work for Event-Driven Data ProductsMeasurement EventsMeasurement Events Often Form Aggregate-Aligned Data ProductsMeasurement Event Sources May Be LossyMeasurement Events May Power Time-Sensitive ApplicationsHybrid Events—State with a Bit of DeltaNotification EventsSummary
8. Bootstrapping Data Products
Getting Started: Bootstrapping with ConnectorsDual WritesPolling the Database to Create Data ProductsChange-Data CaptureChange-Data Capture Using a Transactional OutboxDenormalization and EventificationEventification at the Transactional OutboxEventification in a Dedicated ServiceWhat Should Go In the Event? And What Should Stay Out?Slowly Changing DimensionsBootstrapping Cloud Storage Files to an Event StreamSummary
9. Integrating Event-Driven Data into Data at Rest
Analytics and the Medallion ArchitectureConnecting Event Streams Into Existing Batch-Data FlowsThrough the Lens of Data Mesh: What’s Going On?Through the Lens of Data Mesh: How Do We Solve It?Balancing File Sizes, SLAs, and LatencyBudget Blues: A Tale of OverspendingExtending the Self-Service Platform for Nonstreaming Data ProductsSummary

10. Eventual Consistency
Converging on Consistency, One Event at a TimeStrategies for Dealing with Eventual ConsistencyPrevent Failures to Avoid InconsistencyUse Event-Driven Data Products Instead of Request-Response Server API CallsExpose Eventual Consistency in the Server ResponsePlan for New Services and Reprocessing of DataSynchronize Data Products on Time BoundariesOut-of-Order EventsResolving Late-Arriving EventsSummary
11. Bringing It All Together
Event Streams for Data MeshIntegrating with Existing SystemsOperations, Analytics, and Everything in BetweenSummary
Index
About the Author

Content preview from Building an Event-Driven Data Mesh

Chapter 8. Bootstrapping Data Products

There are two main scenarios for creating event-driven data products. The first involves the creation of data products from existing data sources that are not already in the form of events. As we have alluded throughout the book, these include conventional databases (relational, document, key-value, etc.), cloud filesystems, and even FTP file dumps. In this chapter, we’ll be looking at bootstrapping these existing data sources into the event-driven data mesh.

The second scenario involves data sources that are in event streams, such as data produced by native event-driven services. Since the data source is already event-driven, creating data products tends to be more a function of formalizing what data is emitted and what data should remain concealed within.

Don’t worry too much about getting your first data products exactly right. In fact, it’s best to get some experience under your belt, find what works and what doesn’t work so well, and iterate from there. You can draw a parallel between building data products and building your self-service platform. “Level 1: The Minimal Viable Platform” is a basic but useful platform for getting started with a data mesh that you’ll increment and improve as necessary. Think about your first data products in this very same manner—MVP data products that will start you off on your road to real-time, event-driven processing and get the data available for others to use as they see fit.

In this chapter, we’ll ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098127596Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Building an Event-Driven Data Mesh

by Adam Bellemare

Chapter 8. Bootstrapping Data Products

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.