97 Things Every Data Engineer Should Know

Chapter 3. About the Storage Layer

Julien Le Dem

The main purpose of abstractions in software is to hide complexity. Just as we prove mathematical theorems by using other theorems that have previously been proven, we build software on top of abstraction layers without needing to know exactly how they are implemented. We may understand them; we just don’t need to have every detail in mind when we use them, freeing up our thoughts to concentrate on what we’re trying to achieve.

That said, it is useful (but not necessary) to actually go through these details at least once. Understanding assembly language or compilers makes us better programmers, even if we don’t fiddle with them on a day-to-day basis. The same is true of the storage layer of a database or any data-processing framework.

This storage layer is what provides the familiar two-dimensional table abstraction on top of a linear persistence layer. For example, while writing SQL, we focus on defining constraints that define a result (joins, filters, etc.) without needing to know the data format or layout. The optimizer will figure out an efficient way to produce the correct result.

A naive query engine would load the data in memory and then apply filters and other expressions. Naturally, we want to avoid loading anything that will be discarded as part of query processing. This saves I/O cost as well as CPU cost by avoiding ...