November 2021
Intermediate to advanced
340 pages
10h 46m
Chinese
Content preview from Spark快速大数据分析（第2版）
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
230
第 9 章
用
Apache Spark
构建可靠的数据湖
前面的章节已经介绍了如何简单高效地使用
Spark
来构建可伸缩且性能强大的数据处理流
水线。但在实践中，实现处理逻辑只解决了端到端构建流水线问题的一半。对于数据工程
师、数据科学家或数据分析师来说，构建流水线的终极目标是查询已处理的数据，并从中
获取商业价值。存储解决方案的选择会决定端到端（从原始数据到商业价值）的稳健性和
数据流水线的性能。
本章首先讨论最佳存储解决方案的重要特点，然后讨论数据库和数据湖这两大类存储解决
方案，以及如何在
Spark
中使用
。最后，我们会了解被称为“湖仓一体”的下一代存储解
决方案，并探索这个领域的一些新的开源处理引擎。
9.1
最佳存储解决方案的重要特点
以下是存储解决方案所需要具备的部分特点。
可伸缩性与性能
存储解决方案需要能随数据量伸缩，提供满足工作负载要求的读写吞吐量与延迟。
支持事务
复杂的工作负载经常涉及数据的并发读写，因此支持
ACID
事务对于确保最终结果的质
量至关重要。
支持各种数据格式
存储解决方案应该要能存储非结构化数据（如原始日志这样的文本文件）、半结构化数
据（如
JSON
数据）
，以及结构化数据（如表格数据）。
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.