November 2021
Intermediate to advanced
340 pages
10h 46m
Chinese
Content preview from Spark快速大数据分析（第2版）
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
用
Apache Spark
管理
、
部署与伸缩机器学习流水线
｜
289
境的模型后，这个模型的
URI
是不变的，因此可以用它来获取最新的版本。
# 获取最新的生产模型
model_production_uri = f"models:/{model_name}/production" model_production =
mlflow.spark.load_model(model_production_uri)
11.2.2
流处理
如果不想等每小时或每晚的作业处理数据并生成预测结果，那么可以选择
Structured
Streaming
，它能持续对新输入的数据进行预测
。虽然这种方式比批处理方式更费钱，因为
需要为计算时间持续付费（并且吞吐量更低），但能更及时地生成预测结果，这样就可以
针对预测结果尽快做出反馈。总体来说，流处理解决方案的维护和监控比批处理解决方案
更复杂，但能提供更低的延迟。
使用
Spark
将
批式预测转为流式预测很容易，而且所有代码几乎都一样。唯一的区别是，
读入数据时，需要使用
spark.readStream()
，而不是
spark.read()
，还需要修改读取的数
据源路径。通过流式输入一个含有多个
Parquet
文件的文件夹
，以下示例将模拟读取流式
数据。你会看到，虽然使用
Parquet
文件
，但我们还是声明了表结构。这是因为使用流式
数据时需要提前确定表结构。这个示例会使用基于第
10
章中的
Airbnb
数据集训练出的随
机森林模型进行流式预测。我们会用
MLflow
读取保存的模型
。将源数据分为
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.