Building an Event-Driven Data Mesh

Preface

Data mesh is a fundamental shift in the way we think about, create, share, and use data. We promote data to a first-class citizen by carefully curating and crafting it into data products, supported with the same level of care and commitment as any other business product. Consumers can discover and select the data products they need for their own use cases, relying upon the commitment of the data product producer to maintain and support it. At its heart, data mesh is as much about technological reorganization as it is about the renegotiation of social contracts, responsibilities, and expectations.

Back when I wrote Building Event-Driven Microservices (O’Reilly) I made reference to (and a bit vaguely defined) a data communication layer, very similar yet not nearly so well thought out as data mesh. The principles of the data communication layer were simple enough: treat data as a first-class citizen, make it reliable and trustworthy, and produce it through event streams so that you can power both operational and analytical applications.

The beauty of data mesh is that it’s not a big-bang total revision of everything we know about data. In fact, it’s really an affirmation of best practices, both social and technical, based on the collective hard work and experiences of countless people. It provides the framework necessary to discuss how to go about creating, communicating, and using data, acting as a lingua franca for the data world.

Zhamak Dehghani has done a phenomenal job ...