July 2019
Intermediate to advanced
97 pages
2h 16m
English
Content preview from Rebuilding Reliable Data Pipelines Through Modern Tools
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 6. Identifying Workflow and Pipeline Issues
From its beginning, this book has been about extracting value from your data. That journey to value sometimes requires your data to flow through many different systems and processings. Chapter 5 was all about the failures for a given process. This chapter steps back and looks at the bigger picture, asking how to identify issues at a pipeline level.
Typically, to extract value from data, you need a number of operations to happen first, tasks that include but aren’t limited to the following:
- Join and enrich
-
Joining and enriching the data with external data sources.
- Optimal persist
-
Getting the raw data in the common desired format. This could be storing the data in different storage systems for different optimal access patterns.
- Feature generation
-
Enriching data with logic (human coded or artificial intelligence generated)
- Toxinization/Mask
-
The act of removing personal data and replacing it with universally unique identifiers (UUIDs) to better protect the person the data relates to and to protect the company from legal issues.
- Data marting
-
The action of joining, ordering, or filtering datasets to get them to a more consumable state.
To make things more difficult, the larger an organization is, the more separated these operations will be. Compound that by the fact that you will most likely have hundreds, if not thousands, of datasets with thousands of value-producing approaches and outcomes. That will leave you with many ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.