Streaming Data Mesh

Chapter 6. Self-Service Data Infrastructure

Chapter 1 described self-service infrastructures. We said they are the “easy buttons” that make data product engineering simpler for domains. These easy buttons hide the hard parts behind a streaming data mesh. It is the responsibility of the central team to implement the hard parts behind the easy buttons.

In this chapter, we will list the self-services we will need in a streaming data mesh. We will focus on the names and signature of the self-services (their names and request/response arguments). We will save the discussion on how these self-services will be implemented for Chapter 7.

Self-services are the interfaces (or APIs) that domains see and use when working with a streaming data mesh. There should be a self-service for every function that a domain needs to develop and deploy streaming data products.

Note

The terms service and self-service are used a lot in this book. We use self-service to indicate the interface that faces the domains. The term service refers to any generic service. We will try to avoid referring to self-services as services.

Theself-services described in this chapter will be CLIs that are executed from a terminal. We do this strictly for convenience. These same self-services can be implemented as RESTful services or a web interface. An example CLI can be found in the GitHub repository.

The self-services we define are also designed to ensure that all security requirements, data governance requirements, and ...