Introducing .NET for Apache Spark: Distributed Processing for Massive Datasets

© Ed Elliott 2021

E. ElliottIntroducing .NET for Apache Sparkhttps://doi.org/10.1007/978-1-4842-6992-3_10

10. Troubleshooting

Ed Elliott¹  

(1)

Sussex, UK

 

In this chapter, we will take a look at how to monitor your applications and troubleshoot them. We will look at the log files that you can control and the SparkUI, a web interface for examining Apache Spark jobs, how the jobs ran in terms of performance, and what execution plan your Apache Spark job generated. In this chapter, we will not have any code samples, but we will look at the configuration and the SparkUI web interface.

Logging

Apache Spark uses log4j for its logging, and to control the amount of logging you see, you should look in your spark directory where you will find the “conf” folder, ...