book

Streaming Data Mesh

by Hubert Dulay, Stephen Mooney

May 2023

Intermediate to advanced

223 pages

5h 58m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This BookWhy We Wrote This BookNavigating This BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsHubertStephen
Data DivideData Mesh PillarsData OwnershipData as a ProductFederated Computational Data GovernanceSelf-Service Data PlatformData Mesh DiagramOther Similar Architectural PatternsData FabricData Gateways and Data ServicesData DemocratizationData VirtualizationFocusing on ImplementationApache KafkaAsyncAPI
The Streaming AdvantageStreaming Enables Real-Time Use CasesStreaming Enables Data Optimization AdvantagesReverse ETLThe Kappa ArchitectureLambda Architecture IntroductionKappa Architecture IntroductionSummary
Identifying DomainsDiscernible DomainsGeographic RegionsHybrid ArchitectureMulticloudAvoiding Ambiguous DomainsDomain-Driven DesignDomain ModelDomain LogicBounded ContextThe Ubiquitous LanguageData Mesh Domain RolesData Product EngineerData Product Owner or Data StewardStreaming Data Mesh Tools and Platforms to ConsiderDomain Charge-BacksSummary
Defining Data Product RequirementsIdentifying Data Product DerivativesDerivatives from Other DomainsIngesting Data Product Derivatives with Kafka ConnectConsumabilitySynchronous Data SourcesAsynchronous Data Sources and Change Data CaptureDebezium ConnectorsTransforming Data Derivatives to Data ProductsData StandardizationProtecting Sensitive InformationSQLExtract, Transform, and LoadPublishing Data Products with AsyncAPIRegistering the Streaming Data ProductBuilding an AsyncAPI YAML DocumentAssigning Data TagsVersioningMonitoringSummary
Data Governance in a Streaming Data MeshData Lineage GraphStreaming Data Catalog to Organize Data ProductsMetadataSchemasLineageSecurityScalabilityGenerating the Data Product Page from AsyncAPIApicurio RegistryAccess WorkflowCentralized Versus DecentralizedCentralized EngineersDecentralized (Domain) EngineersSummary
Streaming Data Mesh CLIResource-Related CommandsCluster-Related CommandsTopic-Related CommandsThe domain CommandsThe connect CommandsThe streaming CommandsPublishing a Streaming Data ProductData Governance-Related ServicesSecurity ServicesStandards ServicesLineage ServicesSaaS Services and APIsSummary
InfrastructureTwo Architecture SolutionsDedicated InfrastructureMultitenant InfrastructureStreaming Data Mesh Central ArchitectureThe Domain Agent (aka Sidecar)Data PlaneControl PlaneSummary
The Traditional Data Warehouse StructureIntroducing the Decentralized Team StructureEmpowering PeopleWorking ProcessesFostering CollaborationData-Driven AutomationNew Roles in Data DomainsNew Roles in the Data PlaneNew Roles in Data Science and Business Intelligence
Separating Data Engineering from Data ScienceOnline and Offline Data StoresApache Feast IntroductionSummary

Streaming Data Mesh ExampleDeploying an On-Premises Streaming Data MeshInstalling a ConnectorDeploying Clickstream Connector and Auto-Creating TablesDeploying the Debezium Postgres CDC ConnectorEnrichment of Streaming DataPublishing the Data ProductConsuming Streaming Data ProductsFully Managed SaaS ServicesSummary and Considerations

Content preview from Streaming Data Mesh

Preface

Welcome to this first edition of Streaming Data Mesh! This is your guide to understanding and building a streaming data mesh that meets all of the pillars of a data mesh.

Data mesh is one of the most popular architectures for data platforms that many are exploring today. This book will help you get a full understanding of this self-servicing data platform in a streaming context. Today, batch processing dominates all extract, transform, and load (ETL) processes in most businesses. This book will help show a different perspective of data pipelines and apply the same concepts you already understand in batch ETL, but in a streaming ETL in the context of a data mesh.

This book is designed to help you understand the essential concepts around streaming data mesh—the concepts, architectures, and technologies at its core. The book covers all the essential topics related to streaming mesh, from the basics of data architecture, to the use of big data tools for data warehousing, to business-oriented approaches for streaming data mesh architectures. Additionally, we will look at a stack of services involved in a successful streaming data mesh project.

This book does not require you to have preknowledge of the pillars that make up a data mesh. We will briefly introduce the pillars at a very high level and define them with streaming specifically in mind. If you feel you need to understand data mesh in more detail, please refer to Zhamak Dehghani’s book, Data Mesh (O’Reilly).