Data Management at Scale

Chapter 1. The Disruption of Data Management

Data management is being disrupted because datafication is everywhere. Existing architectures can no longer be scaled up. Enterprises need a new data strategy. A paradigm shift and change of culture are needed too, because the centralized solutions that work today will no longer work in the future.

Technological trends are fragmenting the data landscape. The speed of software delivery is changing with the new methodologies at a cost of increased data complexity. The rapid growth of data and intensive data consumption make operational systems suffer. Lastly, there are privacy, security, and regulatory concerns.

The impact these trends have on data management are tremendous and force the whole industry to rethink how data management must be conducted in the future. In this book, I will lay out a distinctive theory on data management, one that contrasts with how many enterprises have designed and organized their existing data landscape today. Before we come to this in Chapter 2, we need to agree on what data management is, and why it is important. Next, we need to set the scene by looking at different trends. Then, and finally, we will examine how current enterprise data architectures with platforms are designed and organized today.

Before we start, let me lay my cards out on the table. I have strong beliefs about what should be done within data management centrally and what can be done on a federated level. The distributed nature of future ...