book

Data Mesh in Action

by Sven Balnojan, Marian Siwiak, Jacek Majchrzak

February 2023

Intermediate to advanced

328 pages

9h 49m

English

Manning Publications

Read now

Unlock full access

forewordprefaceacknowledgmentsabout this bookWho should read this book?How this book is organized: A road mapHow to use this bookThe Messflix case studyliveBook discussion forumabout the authorsabout the cover illustration
1.1 Data mesh 1011.2 Why the data mesh?1.2.1 Alternatives1.2.2 Data warehouses and data lakes inside the data mesh1.2.3 Data mesh benefits1.3 Use case: A snow-shoveling business1.4 Data mesh principles1.4.1 Domain-oriented decentralized data ownership and architecture1.4.2 Data as a product1.4.3 Federated computational governance1.4.4 Self-serve data infrastructure as a platform1.5 Back to snow shoveling1.6 Socio-technical architecture1.6.1 Conway’s law1.6.2 Team topologies1.6.3 Cognitive load1.7 Data mesh challenges1.7.1 Technological challenges1.7.2 Data management challenges1.7.3 Organizational challengesSummary
2.1 Analyzing data mesh drivers2.1.1 Business drivers2.1.2 Organizational drivers2.1.3 Domain-data drivers2.1.4 Minor organizational drivers2.1.5 Is a data mesh a good fit for me?2.2 Data mesh alternatives and complementary solutions2.2.1 Enterprise data warehouse2.2.2 Data lake2.2.3 Data lakehouse2.2.4 Data fabric2.2.5 Data mesh vs. the rest of the world2.3 Understanding a data mesh implementation effort2.3.1 The data mesh development cycle2.3.2 Development cycle in the shoveling example2.3.3 Enabling the team2.3.4 Development cycle in detailSummary
3.1 Getting the lay of the land3.1.1 Drawing a system landscape diagram3.1.2 Performing stakeholder analysis3.2 Identifying candidates for the MVP implementation team3.2.1 Choosing development teams3.2.2 Choosing the cooperation model3.2.3 Choosing a data governance team3.3 Setting up MVP governance3.3.1 Defining data mesh value statement(s)3.3.2 Defining data governance policies3.3.3 Federating data governance3.4 Developing minimal data products3.4.1 Identifying domain-oriented datasets3.4.2 Choosing data product owners3.4.3 Deciding on the minimum viable data product description3.4.4 Developing the simplest tools to expose your data3.5 Setting up the minimal platform3.5.1 Ensuring platform-forced governability3.5.2 Ensuring platform securitySummary

4.1 Capturing and analyzing domains4.1.1 Domain-driven design 1014.1.2 Invite the right people4.1.3 Choose the correct workshop technique4.2 Applying ownership using domain decomposition4.2.1 Domain, subdomain, and business capability4.2.2 Decompose domains using business capability modeling4.2.3 How are domains and business capabilities related to data?4.2.4 Assign responsibilities to the data-product-owning team4.2.5 Choose the right team to own data4.3 Applying ownership using data use cases4.3.1 Data use cases4.3.2 Model and bounded context4.3.3 Set up boundaries of use-case-driven data products4.3.4 Choose the right team to own data4.4 Applying ownership using design heuristics4.4.1 What is a heuristic?4.4.2 Using design heuristics4.4.3 Designing heuristics and possible boundaries4.5 Final landscape: The mesh of interconnected data products4.5.1 Messflix data mesh4.5.2 Data products form a mesh4.5.3 Is it already a data mesh?Summary
5.1 Applying product thinking5.1.1 Product thinking analysis5.1.2 Data product canvas5.2 What is a data product?5.2.1 Data product definition5.2.2 Product, not project5.2.3 What can be a data product?5.3 Data product ownership5.3.1 Data product owner5.3.2 Data product owner responsibilities5.3.3 An Agile DevOps team as a base for data product dev team5.3.4 Data product owner and product owner5.4 Conceptual architecture of a data product5.4.1 External architecture view5.4.2 Internal architecture view5.5 Data product fundamental characteristics5.5.1 Self-described data product5.5.2 Introduction to metadata5.5.3 Metadata as code5.5.4 Data product metadata5.5.5 Domain dataset metadata5.5.6 Other kinds of metadata5.6 Additional data product characteristics: FAIR and immutability5.6.1 Findability5.6.2 Accessibility5.6.3 Interoperable5.6.4 Reusable5.6.5 Immutable5.7 Data contracts and sharing agreements inside the data mesh5.7.1 Data contracts and sharing agreements5.7.2 Implementing data contracts and sharing agreementsSummary
6.1 Data governance in a nutshell6.2 Benefits of data governance6.2.1 Business value perspective6.2.2 Data usability perspective6.2.3 Data control perspective6.3 Planning data governance outcomes6.3.1 Hierarchy of data governance outcomes6.3.2 Strategic-level outcomes6.3.3 Tactical-level outcomes6.3.4 Implementation-level outcomes6.4 Federating data governance6.4.1 Thinking of data governance in terms of “sliders”6.4.2 Extreme ends of data governance models6.4.3 Federated data governance model6.4.4 Setting-up governance team operations6.5 Making data governance computational6.5.1 Making policies computational6.5.2 Automating policy checksSummary
7.1 The MVP platform7.1.1 Platform definition7.1.2 Platform thinking7.2 Improvements with X as a service7.2.1 X as a service explained7.2.2 X as a service applied7.3 Improvements with platform architecture7.3.1 Platform architecture explained7.3.2 Platform architecture applied7.4 Improvements for the data producersSummary
8.1 Data mesh on Google Cloud Platform8.1.1 Self-serve data platform architecture8.1.2 Identifying the components of the platform8.1.3 Identifying the components of the data product8.1.4 Workflows8.1.5 Variations8.1.6 Relation to data mesh ideas8.1.7 GCP architecture summary8.2 Data mesh on AWS8.2.1 Self-serve data platform architecture8.2.2 Identifying the components of the platform8.2.3 Identifying the components of the data products8.2.4 Workflows8.2.5 Relation to data mesh ideas8.2.6 Variations8.2.7 AWS architecture summary8.3 Data mesh on Databricks8.3.1 Self-serve data platform architecture8.3.2 Identifying the components of the platform8.3.3 Identifying the components of the data product8.3.4 Workflow considerations8.3.5 Variations8.3.6 Databricks architecture summary8.4 Data mesh on Kafka8.4.1 Self-serve data platform architecture8.4.2 Identifying the components8.4.3 Considerations8.4.4 Kafka architecture summarySummary
9.1 Capturing and understanding the current state9.1.1 What is software architecture?9.1.2 How to document architecture: The C4 model9.2 Understanding architectural drivers of a data product design9.2.1 Architectural drivers9.2.2 Capturing architectural drivers for a data-product design9.3 Designing the future architecture of a data product and related systems9.3.1 Design session9.3.2 File-based data product: Spreadsheet9.3.3 From monolith and microservice to a data product9.3.4 Exposing data for stream processing and batch processingSummary
D.1 Notes on thinnest viable platformsD.2 Note on phasing out interfaces

Content preview from Data Mesh in Action

7 The self-serve data platform

This chapter covers

Understanding how our MVP fits into a self-serve data platform
Building a self-serve data platform in three iterations
Applying platform thinking
Exploring X as a service
Working with the modular plane architecture

In the first part of this book, you learned all the basics of the data mesh. That included building a data mesh MVP for Messflix within a month in chapter 3. The data mesh MVP contains a minimal self-serve platform.

After Messflix starts to build data product after data product, a lot of duplication of efforts happens in terms of ingestion tooling and other data-related technologies. These data-related skills are specialized. Thus, once we start to grow the number of data products, ...