Data Mesh

Chapter 12. Design Consuming, Transforming, and Serving Data

A data product’s primary job is to consume data from upstream sources using its input data ports, transform it, and serve the result as permanently accessible data via its output data ports.

In this chapter, I go through the design characteristics of these three basic functions that all data products implement: consume data (“Consume Data”), transform data (“Transform Data”), and serve data (“Serve Data”).

Let’s begin with the one that has the most unique properties to a data mesh approach.

Serve Data

A data product serves domain-oriented data to a diverse set of analytical consumers. It does so through its output data ports (interfaces), introduced in Chapter 9. The output data ports have explicitly defined contracts and APIs.

This seemingly simple affordance of “serving domain-driven data” has interesting properties when considering its relationship with the agents in the ecosystem and their capabilities and needs.

Let’s look at the relationship between a data product and its data users.

The Needs of Data Users

Figure 12-1 shows the needs of data users and how a data product serves them.

Figure 12-1. Serve data properties to satisfy data users and their needs

The requirements driven by data users impose a set of design considerations on how data products serve their data:

Analytical data users have a diverse profile ...