November 2021
Intermediate to advanced
340 pages
10h 46m
Chinese
Content preview from Spark快速大数据分析（第2版）
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
62
｜
第
3
章
# Python代码
row[0]
Out[13]: 350
row[1]
Out[14]: True
row[2]
Out[15]: 'Learning Spark 2E'
有类型对象实际上是
JVM
中的
Java
类对象或
Scala
类对象。
Dataset
中的每个元素都对应
一个
JVM
对象。
3.4.2
创建
Dataset
与从数据源创建
DataFrame
一样，创建
Dataset
时，需要知道表结构。换句话说，你需要
知道数据类型。虽然对于
JSON
格式和
CSV
格式的数据来说
，
Spark
可以自动推断出表结
构，但是对于较大的数据集，这一过程会消耗较多资源。在
Scala
中创建
Dataset
时，
为
Dataset
指定表结构的最简办法是使用样例类
（
case class
）。 在
Java
中，我们使用
JavaBean
类（第
6
章会进一步介绍
JavaBean
和
Scala
样例类）
。
Scala
中的样例类
要想实例化领域专用对象并用于
Dataset
，
可以定义
Scala
的样例类。举个例子，我们来看看
JSON
文件保存的一组物联网设备的读数（稍后本节会在一个完整的示例中使用这个文件）
。
该文件包含多行
JSON
字符串，形式如下所示。
{"device_id": 198164, "device_name": "sensor-pad-198164owomcJZ", "ip":
"80.55.20.25", "cca2": "PL", "cca3": "POL", "cn": "Poland", "latitude":
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.