June 2021
Intermediate to advanced
264 pages
7h 19m
English
Content preview from 97 Things Every Data Engineer Should Know
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 40. Know the Value per Byte of Your Data
Dhruba Borthakur
As a data engineer at various data-driven software companies, I observed how new technologies like Hadoop and Amazon S3 enabled product teams to store a lot of data. Compared to earlier systems, these new systems reduced the cost per byte of data so much that it became economically feasible to store terabytes of data without boring a hole in your pocket. It was easy to calculate this metric: divide your total dataset size by its total cost, and you have your cost-per-byte metric.
Product engineers started to log every event in their applications, without a second thought: “I’ll log fine-grained details about each event, even though I really need only one small piece of this information. It’s cheap to log, so why bother to reduce the size of my log record?”
We data engineers were thrilled to flaunt our terabyte sized datasets, compariing our efforts to those of traditional database administrators who typically managed only up to a few hundred gigabytes of data. General Electric’s locomotives generate 1TB of data in a single freight route. A Boeing 787 generates half a terabyte per flight. And data engineers help manage this data! This was in the mid-2010s, when enterprises leveraged the rapidly diminishing cost per byte to practically never delete their log data (other than for compliance reasons).
Fast-forward to ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.