Building an Event-Driven Data Mesh

Chapter 8. Bootstrapping Data Products

There are two main scenarios for creating event-driven data products. The first involves the creation of data products from existing data sources that are not already in the form of events. As we have alluded throughout the book, these include conventional databases (relational, document, key-value, etc.), cloud filesystems, and even FTP file dumps. In this chapter, we’ll be looking at bootstrapping these existing data sources into the event-driven data mesh.

The second scenario involves data sources that are in event streams, such as data produced by native event-driven services. Since the data source is already event-driven, creating data products tends to be more a function of formalizing what data is emitted and what data should remain concealed within.

Don’t worry too much about getting your first data products exactly right. In fact, it’s best to get some experience under your belt, find what works and what doesn’t work so well, and iterate from there. You can draw a parallel between building data products and building your self-service platform. “Level 1: The Minimal Viable Platform” is a basic but useful platform for getting started with a data mesh that you’ll increment and improve as necessary. Think about your first data products in this very same manner—MVP data products that will start you off on your road to real-time, event-driven processing and get the data available for others to use as they see fit.

In this chapter, we’ll ...