Building an Event-Driven Data Mesh

Chapter 5. Self-Service Data Platform

A self-service data platform makes it easy to discover, use, publish, manage, and secure data and data products for all mesh users. We’ve already introduced some of the components that go into making this platform: the event broker for serving event streams, a metadata store for tracking essential metadata, and access controls for managing data product access. In this chapter, we’ll cover the features of a self-service platform beginning with minimalism and ending with ideal.

There are a few important overarching questions yet to be addressed: How do prospective consumers discover available data products? Once discovered, how do they plug them into their existing applications and analytics? How do they build new applications on top of the data products? Similarly, what does the workflow for a data product publisher look like? How do they go about managing their data products and actually communicating with their customers? The self-service platform must provide a streamlined solution for each of these questions.

The self-service data platform consists partially of components that are already in use in your organization and partially of components that you’ll need to build or buy. For example, you may already have a metadata catalog that tracks important data sets that you can also use for tracking data product metadata. You may be deploying applications using Docker images, running them on Kubernetes, and monitoring them with a cloud-based ...