book

Data Mesh in Action

Name: Data Mesh in Action
ISBN: 9781633439979

by Jacek Majchrzak, Marian Siwiak, Sven Balnojan

February 2023

Intermediate to advanced

328 pages

9h 49m

English

Manning Publications

Read now

Unlock full access

inside front cover
Data Mesh in Action
Copyright
brief contents
contents
front matter
forewordprefaceacknowledgmentsabout this bookWho should read this book?How this book is organized: A road mapHow to use this bookThe Messflix case studyliveBook discussion forumabout the authorsabout the cover illustration
Part 1. Foundations
1 The what and why of the data mesh
1.1 Data mesh 1011.2 Why the data mesh?1.2.1 Alternatives1.2.2 Data warehouses and data lakes inside the data mesh1.2.3 Data mesh benefits1.3 Use case: A snow-shoveling business1.4 Data mesh principles1.4.1 Domain-oriented decentralized data ownership and architecture1.4.2 Data as a product1.4.3 Federated computational governance1.4.4 Self-serve data infrastructure as a platform1.5 Back to snow shoveling1.6 Socio-technical architecture1.6.1 Conway’s law1.6.2 Team topologies1.6.3 Cognitive load1.7 Data mesh challenges1.7.1 Technological challenges1.7.2 Data management challenges1.7.3 Organizational challengesSummary
2 Is a data mesh right for you?
2.1 Analyzing data mesh drivers2.1.1 Business drivers2.1.2 Organizational drivers2.1.3 Domain-data drivers2.1.4 Minor organizational drivers2.1.5 Is a data mesh a good fit for me?2.2 Data mesh alternatives and complementary solutions2.2.1 Enterprise data warehouse2.2.2 Data lake2.2.3 Data lakehouse2.2.4 Data fabric2.2.5 Data mesh vs. the rest of the world2.3 Understanding a data mesh implementation effort2.3.1 The data mesh development cycle2.3.2 Development cycle in the shoveling example2.3.3 Enabling the team2.3.4 Development cycle in detailSummary
3 Kickstart your data mesh MVP in a month
3.1 Getting the lay of the land3.1.1 Drawing a system landscape diagram3.1.2 Performing stakeholder analysis3.2 Identifying candidates for the MVP implementation team3.2.1 Choosing development teams3.2.2 Choosing the cooperation model3.2.3 Choosing a data governance team3.3 Setting up MVP governance3.3.1 Defining data mesh value statement(s)3.3.2 Defining data governance policies3.3.3 Federating data governance3.4 Developing minimal data products3.4.1 Identifying domain-oriented datasets3.4.2 Choosing data product owners3.4.3 Deciding on the minimum viable data product description3.4.4 Developing the simplest tools to expose your data3.5 Setting up the minimal platform3.5.1 Ensuring platform-forced governability3.5.2 Ensuring platform securitySummary

Part 2. The four principles in practice
4 Domain ownership
4.1 Capturing and analyzing domains4.1.1 Domain-driven design 1014.1.2 Invite the right people4.1.3 Choose the correct workshop technique4.2 Applying ownership using domain decomposition4.2.1 Domain, subdomain, and business capability4.2.2 Decompose domains using business capability modeling4.2.3 How are domains and business capabilities related to data?4.2.4 Assign responsibilities to the data-product-owning team4.2.5 Choose the right team to own data4.3 Applying ownership using data use cases4.3.1 Data use cases4.3.2 Model and bounded context4.3.3 Set up boundaries of use-case-driven data products4.3.4 Choose the right team to own data4.4 Applying ownership using design heuristics4.4.1 What is a heuristic?4.4.2 Using design heuristics4.4.3 Designing heuristics and possible boundaries4.5 Final landscape: The mesh of interconnected data products4.5.1 Messflix data mesh4.5.2 Data products form a mesh4.5.3 Is it already a data mesh?Summary
5 Data as a product
5.1 Applying product thinking5.1.1 Product thinking analysis5.1.2 Data product canvas5.2 What is a data product?5.2.1 Data product definition5.2.2 Product, not project5.2.3 What can be a data product?5.3 Data product ownership5.3.1 Data product owner5.3.2 Data product owner responsibilities5.3.3 An Agile DevOps team as a base for data product dev team5.3.4 Data product owner and product owner5.4 Conceptual architecture of a data product5.4.1 External architecture view5.4.2 Internal architecture view5.5 Data product fundamental characteristics5.5.1 Self-described data product5.5.2 Introduction to metadata5.5.3 Metadata as code5.5.4 Data product metadata5.5.5 Domain dataset metadata5.5.6 Other kinds of metadata5.6 Additional data product characteristics: FAIR and immutability5.6.1 Findability5.6.2 Accessibility5.6.3 Interoperable5.6.4 Reusable5.6.5 Immutable5.7 Data contracts and sharing agreements inside the data mesh5.7.1 Data contracts and sharing agreements5.7.2 Implementing data contracts and sharing agreementsSummary
6 Federated computational governance
6.1 Data governance in a nutshell6.2 Benefits of data governance6.2.1 Business value perspective6.2.2 Data usability perspective6.2.3 Data control perspective6.3 Planning data governance outcomes6.3.1 Hierarchy of data governance outcomes6.3.2 Strategic-level outcomes6.3.3 Tactical-level outcomes6.3.4 Implementation-level outcomes6.4 Federating data governance6.4.1 Thinking of data governance in terms of “sliders”6.4.2 Extreme ends of data governance models6.4.3 Federated data governance model6.4.4 Setting-up governance team operations6.5 Making data governance computational6.5.1 Making policies computational6.5.2 Automating policy checksSummary
7 The self-serve data platform
7.1 The MVP platform7.1.1 Platform definition7.1.2 Platform thinking7.2 Improvements with X as a service7.2.1 X as a service explained7.2.2 X as a service applied7.3 Improvements with platform architecture7.3.1 Platform architecture explained7.3.2 Platform architecture applied7.4 Improvements for the data producersSummary
Part 3. Infrastructure and technical architecture
8 Comparing self-serve data platforms
8.1 Data mesh on Google Cloud Platform8.1.1 Self-serve data platform architecture8.1.2 Identifying the components of the platform8.1.3 Identifying the components of the data product8.1.4 Workflows8.1.5 Variations8.1.6 Relation to data mesh ideas8.1.7 GCP architecture summary8.2 Data mesh on AWS8.2.1 Self-serve data platform architecture8.2.2 Identifying the components of the platform8.2.3 Identifying the components of the data products8.2.4 Workflows8.2.5 Relation to data mesh ideas8.2.6 Variations8.2.7 AWS architecture summary8.3 Data mesh on Databricks8.3.1 Self-serve data platform architecture8.3.2 Identifying the components of the platform8.3.3 Identifying the components of the data product8.3.4 Workflow considerations8.3.5 Variations8.3.6 Databricks architecture summary8.4 Data mesh on Kafka8.4.1 Self-serve data platform architecture8.4.2 Identifying the components8.4.3 Considerations8.4.4 Kafka architecture summarySummary
9 Solution architecture design
9.1 Capturing and understanding the current state9.1.1 What is software architecture?9.1.2 How to document architecture: The C4 model9.2 Understanding architectural drivers of a data product design9.2.1 Architectural drivers9.2.2 Capturing architectural drivers for a data-product design9.3 Designing the future architecture of a data product and related systems9.3.1 Design session9.3.2 File-based data product: Spreadsheet9.3.3 From monolith and microservice to a data product9.3.4 Exposing data for stream processing and batch processingSummary
Appendix A.
Appendix B.
Appendix C.
Appendix D.
D.1 Notes on thinnest viable platformsD.2 Note on phasing out interfaces
index
inside back cover

Content preview from Data Mesh in Action

7 The self-serve data platform

This chapter covers

Understanding how our MVP fits into a self-serve data platform
Building a self-serve data platform in three iterations
Applying platform thinking
Exploring X as a service
Working with the modular plane architecture

In the first part of this book, you learned all the basics of the data mesh. That included building a data mesh MVP for Messflix within a month in chapter 3. The data mesh MVP contains a minimal self-serve platform.

After Messflix starts to build data product after data product, a lot of duplication of efforts happens in terms of ingestion tooling and other data-related technologies. These data-related skills are specialized. Thus, once we start to grow the number of data products, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781633439979Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Data Mesh in Action

by Jacek Majchrzak, Marian Siwiak, Sven Balnojan

7 The self-serve data platform

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.