O'Reilly logo

Machine Learning with Spark - Second Edition by Nick Pentreath, Manpreet Singh Ghotra, Rajdeep Dua

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

UI in Spark

Spark provides a web interface which can be used to monitor jobs, see the environment, and run SQL commands.

SparkContext launches a web UI on port 4040 that displays useful information about the application. This includes the following:

  • A list of scheduler stages and tasks
  • A summary of RDD sizes and memory usage
  • Environmental information
  • Information about the running executors

This interface can be accessed by going to http://<driver-node>:4040 in a web browser. If multiple SparkContexts are running on the same host, they will bind to ports beginning with port 4040 (4041, 4042, and so on).

The following screenshots display some of the information provided by the Web UI:

UI showing the Environment of the Spark Content
UI table ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required