November 2021
Intermediate to advanced
340 pages
10h 46m
Chinese
Content preview from Spark快速大数据分析（第2版）
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
160
｜
第
7
章
在显式调用
DataFrame API
的某些方法时，也会创建分区。举例来说，创建一个大型
DataFrame
或者从硬盘上读取大文件时，可以显式控制
Spark
创建指定数量的分区。
// Scala代码
val ds = spark.read.textFile("../README.md").repartition(16)
ds
: org.apache.spark.sql.Dataset
[
String
]
=
[
value: string
]
ds.rdd.getNumPartitions
res5
: Int
= 16
val numDF = spark.range(1000L * 1000 * 1000).repartition(16)
numDF.rdd.getNumPartitions
numDF
: org.apache.spark.sql.Dataset
[
Long
]
=
[
id: bigint
]
res12
: Int
= 16
最后，
混洗分区
会在混洗阶段创建。在默认情况下，混洗分区的分区数为
200
，这是由
spark.sql.shuffle.partitions
决定的。可以根据数据集的大小修改这个值，以减少小分
区的数量，因为这些分区需要跨网络传输到其他执行器以用于任务执行。
spark.sql.shuffle.partitions
的默认值对于小负载作业或流式作业而言太
高了，你可能需要将其调低到执行器核心总数甚至更低。
混洗分区会在执行
groupBy()
或
join()
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.