November 2021
Intermediate to advanced
340 pages
10h 46m
Chinese
Content preview from Spark快速大数据分析（第2版）
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Spark
应用的优化与调优
｜
165
会引发数据在
Spark
执行器间大量移动。
这些转化操作的核心是
Spark
如
何计算出要生成的数据，如何将数据以键和值的对应形式
写入硬盘，如何在
groupBy()
、
join()
、
agg()
、
sortBy()
和
reduceByKey()
等操作中传输
这些键值形式的数据。这种数据移动过程通常被称为
混洗
。
Spark
有
5
种连接算法
，区别主要是跨执行器进行数据交换、移动、排序、分组、合并的
方式。这
5
种算法具体是
：
广播哈希连接
（
broadcast hash join
）、
混洗哈希连接
（
shuffle
hash join
）、
混洗排序合并连接
（
shuffle
sort merge join
）、
广播嵌套循环连接
（
broadcast
nested loop join
）
，以及
混洗
−
复制嵌套循环连接
（
shuffle-and-replicated
nested loop join
，也
称为笛卡儿积连接）。这里，我们只关注最常用的两种算法，即广播哈希连接和混洗排序
合并连接。
7.3.1
广播哈希连接
广播哈希连接也称为
映射端连接
，用于连接一大一小两个数据集，其中小数据集要能在
Spark
驱动器和执行器的内存中放得下
，大数据集可以免于数据移动。这种连接需要基于
某些条件或列。小数据集使用
Spark
广播变量从
Spark
驱动器分发到所有的执行器
，如图
7-6
所示，然后在各执行器上跟大数据集进行连接。这种算法策略避免了大量数据交换。
Spark驱动器
Spark执行器
Spark执行器
较小的数据集
较大的数据集 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.