book

Data Mesh in Action

by Sven Balnojan, Marian Siwiak, Jacek Majchrzak

February 2023

Intermediate to advanced

328 pages

9h 49m

English

Manning Publications

Read now

Unlock full access

forewordprefaceacknowledgmentsabout this bookWho should read this book?How this book is organized: A road mapHow to use this bookThe Messflix case studyliveBook discussion forumabout the authorsabout the cover illustration
1.1 Data mesh 1011.2 Why the data mesh?1.2.1 Alternatives1.2.2 Data warehouses and data lakes inside the data mesh1.2.3 Data mesh benefits1.3 Use case: A snow-shoveling business1.4 Data mesh principles1.4.1 Domain-oriented decentralized data ownership and architecture1.4.2 Data as a product1.4.3 Federated computational governance1.4.4 Self-serve data infrastructure as a platform1.5 Back to snow shoveling1.6 Socio-technical architecture1.6.1 Conway’s law1.6.2 Team topologies1.6.3 Cognitive load1.7 Data mesh challenges1.7.1 Technological challenges1.7.2 Data management challenges1.7.3 Organizational challengesSummary
2.1 Analyzing data mesh drivers2.1.1 Business drivers2.1.2 Organizational drivers2.1.3 Domain-data drivers2.1.4 Minor organizational drivers2.1.5 Is a data mesh a good fit for me?2.2 Data mesh alternatives and complementary solutions2.2.1 Enterprise data warehouse2.2.2 Data lake2.2.3 Data lakehouse2.2.4 Data fabric2.2.5 Data mesh vs. the rest of the world2.3 Understanding a data mesh implementation effort2.3.1 The data mesh development cycle2.3.2 Development cycle in the shoveling example2.3.3 Enabling the team2.3.4 Development cycle in detailSummary
3.1 Getting the lay of the land3.1.1 Drawing a system landscape diagram3.1.2 Performing stakeholder analysis3.2 Identifying candidates for the MVP implementation team3.2.1 Choosing development teams3.2.2 Choosing the cooperation model3.2.3 Choosing a data governance team3.3 Setting up MVP governance3.3.1 Defining data mesh value statement(s)3.3.2 Defining data governance policies3.3.3 Federating data governance3.4 Developing minimal data products3.4.1 Identifying domain-oriented datasets3.4.2 Choosing data product owners3.4.3 Deciding on the minimum viable data product description3.4.4 Developing the simplest tools to expose your data3.5 Setting up the minimal platform3.5.1 Ensuring platform-forced governability3.5.2 Ensuring platform securitySummary

4.1 Capturing and analyzing domains4.1.1 Domain-driven design 1014.1.2 Invite the right people4.1.3 Choose the correct workshop technique4.2 Applying ownership using domain decomposition4.2.1 Domain, subdomain, and business capability4.2.2 Decompose domains using business capability modeling4.2.3 How are domains and business capabilities related to data?4.2.4 Assign responsibilities to the data-product-owning team4.2.5 Choose the right team to own data4.3 Applying ownership using data use cases4.3.1 Data use cases4.3.2 Model and bounded context4.3.3 Set up boundaries of use-case-driven data products4.3.4 Choose the right team to own data4.4 Applying ownership using design heuristics4.4.1 What is a heuristic?4.4.2 Using design heuristics4.4.3 Designing heuristics and possible boundaries4.5 Final landscape: The mesh of interconnected data products4.5.1 Messflix data mesh4.5.2 Data products form a mesh4.5.3 Is it already a data mesh?Summary
5.1 Applying product thinking5.1.1 Product thinking analysis5.1.2 Data product canvas5.2 What is a data product?5.2.1 Data product definition5.2.2 Product, not project5.2.3 What can be a data product?5.3 Data product ownership5.3.1 Data product owner5.3.2 Data product owner responsibilities5.3.3 An Agile DevOps team as a base for data product dev team5.3.4 Data product owner and product owner5.4 Conceptual architecture of a data product5.4.1 External architecture view5.4.2 Internal architecture view5.5 Data product fundamental characteristics5.5.1 Self-described data product5.5.2 Introduction to metadata5.5.3 Metadata as code5.5.4 Data product metadata5.5.5 Domain dataset metadata5.5.6 Other kinds of metadata5.6 Additional data product characteristics: FAIR and immutability5.6.1 Findability5.6.2 Accessibility5.6.3 Interoperable5.6.4 Reusable5.6.5 Immutable5.7 Data contracts and sharing agreements inside the data mesh5.7.1 Data contracts and sharing agreements5.7.2 Implementing data contracts and sharing agreementsSummary
6.1 Data governance in a nutshell6.2 Benefits of data governance6.2.1 Business value perspective6.2.2 Data usability perspective6.2.3 Data control perspective6.3 Planning data governance outcomes6.3.1 Hierarchy of data governance outcomes6.3.2 Strategic-level outcomes6.3.3 Tactical-level outcomes6.3.4 Implementation-level outcomes6.4 Federating data governance6.4.1 Thinking of data governance in terms of “sliders”6.4.2 Extreme ends of data governance models6.4.3 Federated data governance model6.4.4 Setting-up governance team operations6.5 Making data governance computational6.5.1 Making policies computational6.5.2 Automating policy checksSummary
7.1 The MVP platform7.1.1 Platform definition7.1.2 Platform thinking7.2 Improvements with X as a service7.2.1 X as a service explained7.2.2 X as a service applied7.3 Improvements with platform architecture7.3.1 Platform architecture explained7.3.2 Platform architecture applied7.4 Improvements for the data producersSummary
8.1 Data mesh on Google Cloud Platform8.1.1 Self-serve data platform architecture8.1.2 Identifying the components of the platform8.1.3 Identifying the components of the data product8.1.4 Workflows8.1.5 Variations8.1.6 Relation to data mesh ideas8.1.7 GCP architecture summary8.2 Data mesh on AWS8.2.1 Self-serve data platform architecture8.2.2 Identifying the components of the platform8.2.3 Identifying the components of the data products8.2.4 Workflows8.2.5 Relation to data mesh ideas8.2.6 Variations8.2.7 AWS architecture summary8.3 Data mesh on Databricks8.3.1 Self-serve data platform architecture8.3.2 Identifying the components of the platform8.3.3 Identifying the components of the data product8.3.4 Workflow considerations8.3.5 Variations8.3.6 Databricks architecture summary8.4 Data mesh on Kafka8.4.1 Self-serve data platform architecture8.4.2 Identifying the components8.4.3 Considerations8.4.4 Kafka architecture summarySummary
9.1 Capturing and understanding the current state9.1.1 What is software architecture?9.1.2 How to document architecture: The C4 model9.2 Understanding architectural drivers of a data product design9.2.1 Architectural drivers9.2.2 Capturing architectural drivers for a data-product design9.3 Designing the future architecture of a data product and related systems9.3.1 Design session9.3.2 File-based data product: Spreadsheet9.3.3 From monolith and microservice to a data product9.3.4 Exposing data for stream processing and batch processingSummary
D.1 Notes on thinnest viable platformsD.2 Note on phasing out interfaces

Content preview from Data Mesh in Action

3 Kickstart your data mesh MVP in a month

This chapter covers

Selecting part of your organization for building your first data products
Choosing the right people to work on your project
Setting up the governance of your data mesh
Discovering and describing your first data product
Creating the minimal viable infrastructure as a platform (aka the central platform)

In chapter 1, you learned what a data mesh is and what its elements are. In chapter 2, we provided a framework for analysis in terms of whether a data mesh is suitable for you and your business. If you’re here, it means you’re ready to put that knowledge into practice.

You may use this chapter as a learn-as-you-go experience and, following your Messflix alter-ego example, build your ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Data Mesh

Zhamak Dehghani

Data Lakehouse in Action

Pradeep Menon

Streaming Data Mesh

Hubert Dulay, Stephen Mooney

Implementing Data Mesh

Jean-Georges Perrin, Eric Broda

Publisher Resources

ISBN: 9781633439979Publisher Support Other Publisher Website Purchase Link