Rebuilding Reliable Data Pipelines Through Modern Tools

Chapter 7. Watching and Learning from Your Jobs

The whole rationale behind big data and data processing is the belief that having data gives us insights that allow us to make better decisions. However, sometimes organizations dedicated to analyzing data don’t make the extra effort to collect data about their acting on the data.

This data about how we process data can allow you to discover game-changing insights to improve your operations, reliability, and resource utilization. The next chapter digs down into different insights we can get from our data, but before we dig into the outputs, this chapter focuses on the inputs.

This chapter also talks about different types of data that we can collect on our data processing and data storage ecosystem that can help us improve our processes and manage our workloads better.

But before we dig into the different metrics that we want to collect, let’s talk about how you can go about collecting this data and changing the culture of collecting data of workflows.

Culture Considerations of Collecting Data Processing Metrics

We all know that having metrics is a good thing, and yet so few of us gather them from our data processing workloads and data pipelines. Below are some key tips for changing the culture to encourage increased metric coverage.

Make It Piece by Piece

As you read through the upcoming list of data, try not to become overwhelmed if you are not currently collecting all this data. If you are not even collecting a majority of what ...