November 2021
Intermediate to advanced
340 pages
10h 46m
Chinese
Content preview from Spark快速大数据分析（第2版）
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Spark SQL
与
DataFrame
：
读写外部数据源
｜
115
我们用一个示例来帮助你理解这些属性的用法。假设使用如下设置。
•
numPartitions: 10
•
lowerBound: 1000
•
upperBound: 10000
分区步长为
1000
，共有
10
个分区。这与分别执行以下
10
条查询等价（每条对应一个分区）。
•
SELECT * FROM table WHERE partitionColumn BETWEEN 1000 and 2000
•
SELECT * FROM table WHERE partitionColumn BETWEEN 2000 and 3000
•
...
•
SELECT * FROM table WHERE partitionColumn BETWEEN 9000 and 10000
以下是设置这些连接属性时的一些注意点，虽然不是面面俱到，但还是有必要牢记在心。
•
选择
numPartitions
的取值时，最好先使用
Spark
工作节点数量的整数倍。如果有
4
个
工作节点，那么可以从
4
或
8
个分区开始测试
。源系统处理读请求的能力也是关键因素
之一。对于有处理窗口的系统，可以将源系统的请求并发度最大化；对于没有处理窗口
的系统（比如连续处理数据的
OL
TP
系统），应该减少并发请求的数量，以避免耗尽源
系统的资源。
•
一开始，
lowerBound
和
upperBound
的取值最好根据
partitionColumn
的
实际
最大值和最
小值来确定。如果设置了
{numPartitions:10, ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.