June 2021
Intermediate to advanced
264 pages
7h 19m
English
Content preview from 97 Things Every Data Engineer Should Know
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 16. Data Engineering != Spark
Jesse Anderson
The misconception that Apache Spark is all you’ll need for your data pipeline is common.1 The reality is that you’re going to need components from three general types of technologies to create a data pipeline. These three general types of big data technologies are:
Computation
Storage
Messaging
Fixing and remedying this misconception is crucial to success with big data projects or one’s own learning about big data. Spark is just one part of a larger big data ecosystem that’s necessary to create data pipelines.
Put another way:
- Data Engineering = Computation + Storage + Messaging + Coding + Architecture + Domain Knowledge + Use Cases
Batch and Real-Time Systems
Generally, you have to solve two core problems in a batch data pipeline. The first is computation, and the second is the storage of data. Spark is a good solution for handling batch computation, but it can be more difficult to find the right storage sollution—or more correctly, too identify the different and optimized storage technologies for your use case.
Computation Component
Computation is how your data gets processed. Computation frameworks are responsible for running the algorithms and the majority of your code. For big data, they’re responsible for resource allocation, running the code in a distributed fashion, and persisting the results.
Storage Component ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.