November 2021
Intermediate to advanced
340 pages
10h 46m
Chinese
Content preview from Spark快速大数据分析（第2版）
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
92
｜
第
4
章
表4-4：
DataFrameReader
和
DataFrameWriter
中的CSV选项
属 性 名 合 法 值 含 义 作用范围
compression
none
、
bzip2
、
deflate
、
gzip
、
lz4
、
snappy
使用指定压缩编码写数据 写
dateFormat
yyyy-MM-dd
等日期时间
格式
使用
Java
的
DateTimeFormatter
支持的任何格式 读
/
写
multiLine
true
、
false
使用多行模式。默认为
false
（单行模式） 读
inferSchema
true
、
false
如果为
true
，那么
Spark
会自动确定字段的数据类型。
默认为
false
读
sep
任意字符 用该字符分隔一行中各列的值。默认使用逗号（
,
）作
为分隔符
读
/
写
escape
任意字符 用该字符转义引号，默认为
\
读
/
写
header
true
、
false
表明第一行是否为各字段名组成的表头。默认为
false
读
/
写
4.3.6
Avro
Spark 2.4
引入了对
Avro
格式的支持。例如，在序列化和反序列化消息时，
Apache Kafka
就会使用
A
vro
格式。
Avro
格式有很多优点，包括直接映射
JSON
、快速且高效、支持多种
编程语言。
1.
将
Avro
文件读入
DataFrame
用
DataFrameReader
将
Avro
文件读入
DataFrame
与本节介绍的其他数据源的用法一致。
// Scala代码
val
df = spark.read.format("avro")
.load("/databricks-datasets/learning-spark-v2/flights/summary-data/avro/*") ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.