November 2021
Intermediate to advanced
340 pages
10h 46m
Chinese
Content preview from Spark快速大数据分析（第2版）
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
148
｜
第
6
章
Encoder<UsageCost> usageCostEncoder = Encoders.bean(UsageCost.class);
如果是用
Scala
，那么
Spark
会自动生成用于高效转化的字节码。接下来，我们将介绍
Spark
内部的
T
ungsten
格式。
6.4.1
Spark
内部格式与
Java
对象格式
Java
对象有很多额外开销，比如头信息、哈希码、
Unicode
信息等。即使是像
"abcd"
这样
简单的
Java
字符串
，也会占
48
字节的存储空间，而不是表面上看起来的
4
字节。想想创
建一个
MyClass(Int, String, String)
对象的额外开销有多大吧。
Spark
不为
Dataset
或
DataFrame
创建基于
JVM
的对象
，而会分配
Java
堆外内存来存放数
据，并使用编码器将内存表示的数据转为
JVM
对象
。图
6-1
展示了
JVM
对象
MyClass(Int,
String, String)
是如何在
Spark
内部存储的。
JVM对象
内部表示
图 6-1：JVM 对象以紧挨在一起的方式存储在 Spark 管理的 Java 堆外内存中
当数据以这种紧挨在一起的方式存储并且通过指针和偏移量访问时，编码器可以快速序列
化或反序列化数据。这是什么意思呢？
6.4.2
序列化与反序列化
序列化和反序列化不是分布式计算提出的新概念。在分布式计算中，数据频繁地跨网络在
集群内的计算节点间传输。序列化是指发送方将有类型对象编码为二进制格式的过程；反
序列化则是指接收方将数据从二进制格式解码成相应数据类型的对象的过程。 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.