In this chapter, we will take a look at how to monitor your applications and troubleshoot them. We will look at the log files that you can control and the SparkUI, a web interface for examining Apache Spark jobs, how the jobs ran in terms of performance, and what execution plan your Apache Spark job generated. In this chapter, we will not have any code samples, but we will look at the configuration and the SparkUI web interface.
Logging
Apache Spark uses log4j for its logging, and to control the amount of logging you see, you should look in your spark directory where you will find the “conf” folder, ...
