July 2019
Intermediate to advanced
97 pages
2h 16m
English
Content preview from Rebuilding Reliable Data Pipelines Through Modern Tools
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 7. Watching and Learning from Your Jobs
The whole rationale behind big data and data processing is the belief that having data gives us insights that allow us to make better decisions. However, sometimes organizations dedicated to analyzing data don’t make the extra effort to collect data about their acting on the data.
This data about how we process data can allow you to discover game-changing insights to improve your operations, reliability, and resource utilization. The next chapter digs down into different insights we can get from our data, but before we dig into the outputs, this chapter focuses on the inputs.
This chapter also talks about different types of data that we can collect on our data processing and data storage ecosystem that can help us improve our processes and manage our workloads better.
But before we dig into the different metrics that we want to collect, let’s talk about how you can go about collecting this data and changing the culture of collecting data of workflows.
Culture Considerations of Collecting Data Processing Metrics
We all know that having metrics is a good thing, and yet so few of us gather them from our data processing workloads and data pipelines. Below are some key tips for changing the culture to encourage increased metric coverage.
Make It Piece by Piece
As you read through the upcoming list of data, try not to become overwhelmed if you are not currently collecting all this data. If you are not even collecting a majority of what ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.