O'Reilly logo

Learning Real-time Processing with Spark Streaming by Sumit Gupta

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Monitoring streaming jobs

Spark provides different ways to monitor our Spark Streaming job. Though monitoring also depends upon the underlying cluster manager, at the same time, Spark itself produces a lot of information about our jobs that can be captured, analyzed, and further help to monitor and tune our jobs:

Monitoring streaming jobs

The preceding illustration shows the various monitoring options available with Spark, by default. At high level, the monitoring options can be categorized into two different categories, either of which can be leveraged to monitor our streaming jobs deployed in the standalone mode.

Application or job UI

Spark hosts a Web-UI over HTTP (by default, ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required