November 2021
Intermediate to advanced
340 pages
10h 46m
Chinese
Content preview from Spark快速大数据分析（第2版）
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
196
｜
第
8
章
许多常用的监控框架（如
Ganglia
、
Graphite
等）。由于汇报的数据量比较大，因此在默认
情况下，
Structured
Streaming
查询不会启用这些指标。要想启用这些指标，除了为
Spark
配置
Dropwizard
Metrics
，还需要在启动查询前将
SparkSession
的配置参数
spark.sql.
streaming.metricsEnabled
设置为
true
。
注意，在通过
StreamingQuery.lastProgress()
可以获取的信息中，只有一部分信息会通过
Dropwizard Metrics
发布。如果想向任意位置持续发布更多进度信息，需要实现自定义监听
器，接下来介绍具体怎么做。
3.
用自定义
StreamingQueryListener
发布指标
StreamingQueryListener
是一个事件监听器接口，用户可以通过这个接口注入任意逻辑来
持续发布指标。这个接口只可在
Scala
或
Java
中使用。使用自定义监听器分两步。
1.
实现自定义监听器
。
StreamingQueryListener
接口提供了
3
个可供自行实现的方法，以
分别获取流式查询中的
3
类事件
：启动、进展（比如有了一次触发执行），以及终止。
示例如下。
// Scala代码
import org.apache.spark.sql.streaming._
val
myListener
= new StreamingQueryListener
() {
override def
onQueryStarted(event ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.