book

Streaming Data Mesh

by Hubert Dulay, Stephen Mooney

May 2023

Intermediate to advanced

223 pages

5h 58m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This BookWhy We Wrote This BookNavigating This BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsHubertStephen
Data DivideData Mesh PillarsData OwnershipData as a ProductFederated Computational Data GovernanceSelf-Service Data PlatformData Mesh DiagramOther Similar Architectural PatternsData FabricData Gateways and Data ServicesData DemocratizationData VirtualizationFocusing on ImplementationApache KafkaAsyncAPI
The Streaming AdvantageStreaming Enables Real-Time Use CasesStreaming Enables Data Optimization AdvantagesReverse ETLThe Kappa ArchitectureLambda Architecture IntroductionKappa Architecture IntroductionSummary
Identifying DomainsDiscernible DomainsGeographic RegionsHybrid ArchitectureMulticloudAvoiding Ambiguous DomainsDomain-Driven DesignDomain ModelDomain LogicBounded ContextThe Ubiquitous LanguageData Mesh Domain RolesData Product EngineerData Product Owner or Data StewardStreaming Data Mesh Tools and Platforms to ConsiderDomain Charge-BacksSummary
Defining Data Product RequirementsIdentifying Data Product DerivativesDerivatives from Other DomainsIngesting Data Product Derivatives with Kafka ConnectConsumabilitySynchronous Data SourcesAsynchronous Data Sources and Change Data CaptureDebezium ConnectorsTransforming Data Derivatives to Data ProductsData StandardizationProtecting Sensitive InformationSQLExtract, Transform, and LoadPublishing Data Products with AsyncAPIRegistering the Streaming Data ProductBuilding an AsyncAPI YAML DocumentAssigning Data TagsVersioningMonitoringSummary
Data Governance in a Streaming Data MeshData Lineage GraphStreaming Data Catalog to Organize Data ProductsMetadataSchemasLineageSecurityScalabilityGenerating the Data Product Page from AsyncAPIApicurio RegistryAccess WorkflowCentralized Versus DecentralizedCentralized EngineersDecentralized (Domain) EngineersSummary
Streaming Data Mesh CLIResource-Related CommandsCluster-Related CommandsTopic-Related CommandsThe domain CommandsThe connect CommandsThe streaming CommandsPublishing a Streaming Data ProductData Governance-Related ServicesSecurity ServicesStandards ServicesLineage ServicesSaaS Services and APIsSummary
InfrastructureTwo Architecture SolutionsDedicated InfrastructureMultitenant InfrastructureStreaming Data Mesh Central ArchitectureThe Domain Agent (aka Sidecar)Data PlaneControl PlaneSummary
The Traditional Data Warehouse StructureIntroducing the Decentralized Team StructureEmpowering PeopleWorking ProcessesFostering CollaborationData-Driven AutomationNew Roles in Data DomainsNew Roles in the Data PlaneNew Roles in Data Science and Business Intelligence
Separating Data Engineering from Data ScienceOnline and Offline Data StoresApache Feast IntroductionSummary

Streaming Data Mesh ExampleDeploying an On-Premises Streaming Data MeshInstalling a ConnectorDeploying Clickstream Connector and Auto-Creating TablesDeploying the Debezium Postgres CDC ConnectorEnrichment of Streaming DataPublishing the Data ProductConsuming Streaming Data ProductsFully Managed SaaS ServicesSummary and Considerations

Content preview from Streaming Data Mesh

About the Authors

Hubert Dulay is a developer relations and data engineer at StarTree. He is a veteran engineer with over 20 years of experience in big and fast data and MLOps. Hubert has compiled his experiences with data from his time while consulting for many financial institutions, health-care organizations, and telecommunications companies, providing simple solutions that solved many data problems.

Stephen Mooney is an independent data scientist and data engineer serving multiple clients. With over 20 years of experience in big data, MLOps, and data science, he has worked in many major companies across health care, retail, and the public sector. Through this experience, Stephen has delivered many technical and functional projects throughout the entire product lifecycle.