Data Mesh

Chapter 8. Before the Inflection Point

Today’s problems come from yesterday’s “solutions.”

Peter M. Senge, The Fifth Discipline

Organizational complexity, proliferation of data sources, growth in data expectations: these are the forces that have put stress on our existing approaches to analytical data management. Our existing methods have made remarkable progress scaling the machines: managing large volumes of a variety of data types with planet-scale distributed data storage, reliably transmitting high-velocity data through streams, and processing data-intensive workloads concurrently and fast. However, our methods have limitations with regard to organizational complexity and scale, the human scale.

In this chapter, I briefly introduce the current landscape of data architectures, their underlying characteristics, and the reasons why, moving into the future, they limit us.

Evolution of Analytical Data Architectures

How we manage analytical data has gone through evolutionary changes, changes driven by new consumption models, ranging from traditional analytics in support of business decisions to intelligent products augmented with ML. While we have seen an accelerated growth in the number of analytical data technologies, the high-level architecture has seen very few changes. Let’s browse the high-level analytical data architectures, followed by a review of their unchanged characteristics.

Note

The underlying technologies supporting each of the following architectural paradigms ...