Data Mesh

Part IV. How to Design the Data Product Architecture

The data product is a central concept to data mesh. Architecturally, it is designed as an architecture quantum and called a data quantum. A data quantum encapsulates and implements all the necessary behavior and structural components to process and share data as a product (Chapter 3). It builds and runs with autonomy yet connects to other data quanta on the mesh. Interconnection of data quanta creates the symmetric and scale-out architecture of data mesh.

All data products share a set of common properties; e.g., they consume data from upstream sources, transform data, serve data, govern data, etc. This part of the book discusses an opinionated way of designing each of these properties.

I have organized this part of the book around the individual affordances¹ of a data product—the relationship between properties of a data product and how people (or systems) can interact with them. For example, how data mesh users discover, read, or manage the life cycle of data products, directly interacting with one.

Chapter 11, “Design a Data Product by Affordances”, summarizes the approach to the design of a data product. Chapter 12, “Design Consuming, Transforming, and Serving Data”, discusses how data products afford consuming, transforming, and serving data for a diverse set of users—programs and people. It discusses a set of necessary design constraints to make a distributed system of data sharing work. Chapter 13, “Design ...