Data Mesh in Practice

Chapter 1. Pain Points of Centralized Data Responsibility

In the first decade of this century, there was a strong trend toward centralization of enterprise data: all the transactional data of a company needed for its IT operations was consolidated in one central monolithic database in order to have a single source of truth. In the operational data field, architects started to move away from the paradigm of having one central place of truth for transactional data. Since the rise of NoSQL and microservices, it has been a standard approach for individual services to maintain their own data store.

In the analytics data field, however, the centralization paradigm is still prevalent, in terms of both using centralized storage systems and maintaining central data teams that centralize data expertise. Analytical data is often used by overarching functions such as finance, management, or marketing, who need a holistic view of the entire business. The industry went through several iterations of analytical data architectures that innovated on how the data is physically stored and how it is processed. Yet, seemingly because of this need for a holistic view of data, no one dared to touch the general paradigm of central analytics data ownership as the gold standard to gain reliable insights about the business. In the next sections, we look at the currently most used analytics data approaches, their differences, and their commonalities.

The Data Warehouse Approach

The data warehouse approach ...