This book began with a very high-level overview of data processing history and then descended to a more detailed review of the different components of a data ecosystem. We are now diving to a low enough altitude that we will look into the foundations of all data processing at a job level.
The goal of this chapter is to lay a foundation for you to build on later in the book when we begin categorizing different systems and discussing issues that frequently come up in data processing systems. This book is about optimizing the entire pipeline which you can’t do without understanding what is happening in its smallest units.
Then, we ascend a bit and look at Directed Acyclic Graphs (DAGs) at a pipeline level, which are DAGs that make up many processing DAGs and storage systems.
What Is a DAG?
The first time I heard about a DAG engine, I was super hopeful it was going to be groundbreaking and would change my understanding of data processing. If you have a name as cool as a DAG engine, by law you need to be awesome. However, as with many things in life, as you learn the details, reality becomes less exciting than we had hoped.
A DAG is nothing more than a graph of nodes and edges in which you don’t loop back, as shown in Figure 4-1.
Figure 4-1. Simple DAG
The DAG provides a great way to explain all data processing and even data pipelines. In general, ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month, and much more.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
Julian F.
Head of Cybersecurity
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.
Addison B.
Field Engineer
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.
Amir M.
Data Platform Tech Lead
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
