June 2021
Intermediate to advanced
264 pages
7h 19m
English
Content preview from 97 Things Every Data Engineer Should Know
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 68. The Dos and Don’ts of Data Engineering
Christopher Bergh
Here are the things that I wish I had known when I started out in the data industry years ago.
Don’t Be a Hero
Data analytics teams work long hours to compensate for the gap between performance and expectations. When a deliverable is met, the data analytics team are considered heroes. Everyone loves to get awards at the company meeting; however, heroism is a trap.
Heroes give up work/life balance. Yesterday’s heroes are quickly forgotten when there is a new crisis or deliverable to meet. The long hours eventually lead to burnout, anxiety, and even depression. Heroism is difficult to sustain over a long period of time, and it ultimately just reinforces the false belief that the resources and methodologies in place are adequate.
Don’t Rely on Hope
When a deadline must be met, it is tempting to quickly produce a solution with minimal testing, push it out to the users, and hope it does not break. This approach has inherent risks. Eventually, a deliverable will contain errors, upsetting users and harming the hard-won credibility of the data analytics team.
Many data professionals will recognize this situation. You work late on Friday night to get high-priority changes completed. After a heroic effort, you get it done and go home. Saturday morning you wake up, startled. “I forgot ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.