July 2019
Intermediate to advanced
97 pages
2h 16m
English
Content preview from Rebuilding Reliable Data Pipelines Through Modern Tools
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 8. Closing Thoughts
If you have made it this far, thank you. I tried very hard to fit a lot of content into a small package and deliver it in a short time frame. I’m sure there might be areas where you feel as if questions were not answered or detail was missing. It would be nearly impossible to reach every angle of data processing and pipelines in a 90-page book.
I want to leave you with the following key points:
- A better understanding of expectations and their origins
-
The early chapters looked at how many current problems stem from Excel and the mountain of expectations we are still trying to reach today.
- A better understanding of the evolution of tech
-
From the single node, to distributed, to the cloud, so much has changed. And data has only become bigger, more complex, and more difficult to understand.
- A different perspective of the data organization’s different parts
-
The metaphor of the chef, the oven, and the refrigerator shows how meta management, data processing, and storage need to play together in a carefully orchestrated dance in order to make anything work.
- A deeper viewpoint on the complexity of data processing jobs
-
This book went deep into DAGs, shuffles, and bottlenecks. The aim was to help you understand why and how problems pop up in distributed processing.
- A scary look into all the ways things can fail
-
Understanding and expecting failure is the first step to a more maintainable life in development. I hope that the last couple of chapters instilled ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.