November 2021
Intermediate to advanced
340 pages
10h 46m
Chinese
Content preview from Spark快速大数据分析（第2版）
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
18
｜
第
2
章
$
cd spark-3.0.0-preview2-bin-hadoop2.7
$
ls
LICENSE R RELEASE conf examples kubernetes python yarn
NOTICE README.md bin data jars licenses sbin
接下来我们简单总结一下其中部分文件和目录的内容和用途。
Spark 2.
x
和
Spark 3.0
加入了
不少新特性，之前已有的一些文件和目录也有了新的变化。
README.md
这个文件包含如何使用
Spark shell
、如
何从源码编译
Spark
、如何运行独立的
Spark
示
例程序，以及
Spark
文档和配置指南的具体链接，另外还有如何向
Spark
贡献的说明。
bin
顾名思义，这个目录包含与
Spark
交互需要用到的大多数脚本
，包括各种
Spark shell
（
spark-sql
、
pyspark
、
spark-shell
，以及
sparkR
）。本章稍后将使用该目录下的这些
shell
和可执行文件
。具体来说，我们将使用
spark-submit
来提交独立的
Spark
应用，
并在基于
Kubernetes
支持运行
Spark
时编写脚本来构建和推送
Docker
镜像。
sbin
这个目录下的大部分脚本是用于管理的，比如在各种部署模式下启动或停止集群的
Spark
组件。关于部署模式的详细介绍，参见表
1-1
。
kubernetes
从
Spark 2.4
开始
，这个目录包含在
Kubernetes
集群中为
Spark
分发构建
Docker
镜像 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.