Rebuilding Reliable Data Pipelines Through Modern Tools

Chapter 3. The Data Ecosystem Landscape

This chapter focuses on defining the different components of today’s data ecosystem environments. The goal is to provide context for how our problem of data processing fits within the data ecosystem as a whole.

The Chef, the Refrigerator, and the Oven

In general, all modern data ecosystems can be divided into three metaphorical groups of functionality and offerings:

Chef

Responsible for design and metamanagement. This is the mind behind the kitchen. This person decides what food is bought and by what means it should be delivered. In modern kitchens the chef might not actually do any cooking. In the data ecosystem world, the chef is most like design-time decisions and a management layer for all that is happening in the kitchen.

Refrigerator

Handles publishing and persistence. This is where food is stored. It has preoptimized storage structures for fruit, meat, vegetables, and liquids. Although the chef is the brains of the kitchen, the options for storage are given to the chef. The chef doesn’t redesign a different refrigerator every day. The job of the fridge is like the data storage layer in our data ecosystem: keep the data safe and optimized for access when needed.

Oven

Deals with access and processing. The oven is the tool in which food from the fridge is processed to make quality meals while producing value. In this relation, the oven is an example of the processing layer in the data ecosystem, like SQL; Extract, Transform, and ...