November 2021
Intermediate to advanced
340 pages
10h 46m
Chinese
Content preview from Spark快速大数据分析（第2版）
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
152
第 7 章
Spark
应用的优化与调优
第
6
章详细介绍了如何在
Java
和
Scala
中操作
Dataset
。我们探索了
Spark
如何管理内存来
存放
Dataset
这种高级
API
的数据结构
，还分析了与使用
Dataset
相关的一些开销，以及如
何减少这些开销。
除了减少开销，我们也要思考如何优化和调优
Spark
。本章将讨论
Spark
的一些可优化参
数，介绍
Spark
的各种连接策略，查看
Spark
的
UI
，并寻找性能不佳的线索。
7.1
为效率而优化与调优
Spark
有很多可调优的配置参数，本书只会介绍一些最重要和最常调优的配置。你可以通
过查阅文档来获取按功能分组的详细配置列表。
7.1.1
查看与设置
Spark
配置
获取和设置
Spark
配置属性的方式有三种。第一种方式是通过一组配置文件。在安装环
境的
$SPARK_HOME
目录（安装
Spark
的地方）下有一组配置文件：
conf/spark-defaults.conf.
template
、
conf/log4j.properties.template
，以及
conf/spark-env
.sh.template
。修改这些文件中
的默认值，并将其保存为没有
.template
后缀的文件，
Spark
就会使用这些新配置的值。
修改
conf/spark-defaults.conf
文件中的配置参数，这种方式适用于
Spark
集
群，以及所有提交到集群的
Spark
应用。
Spark
应用的优化与调优
｜
153
第二种方式是直接在
Spark
应用内指定配置参数，或者在用
spark-submit ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.