The Informed Company

Chapter EighteenModern versus Traditional Data Stacks: What's Changed?

What's Changed?

This book intentionally outlines how to create a modern stack in a straightforward way, but for those who've been in data for years, some of it may be controversial. Many of these strategies will be known to you, but few have been fully published. We want to broaden the conversation on data modeling and orient the collective goal of analytics toward helping others leverage data for insights.

Many things have changed in the data space that has allowed us to become less technologically constrained and more user‐focused. The main driving forces of change in data are increasing volume, demand, and user base, coupled with the great decrease in cost.

Storage and Compute Continues to Drop in Price Rapidly

Costs have been plummeting for storing and computing data. The main things driving down these costs are breakthroughs in both architecture and hardware over the past few decades.

Architecture

• – Cloud—things are cheap and easily accessed and scaled
• – C‐Store—column‐oriented databases
• – Massively Parallel Processing (MPP)
• – Separation of storage and compute

Hardware

• – SSD
• – Vectorized processing (GPUs and now CPUs)
• – Continued lowering of storage and compute hardware costs

These advances have made optimizing for storage and performance not as valuable anymore.

Data Lakes Were Added to the Stack and ELT Replaced ETL

Data lakes are a stable stopping point and much less prone to errors. Also ...