Scala and Spark for Big Data Analytics

Debugging Spark jobs running as local and standalone mode

While debugging your Spark application locally or as standalone mode, you should know that debugging the driver program and debugging one of the executors is different since using these two types of nodes requires different submission parameters passed to spark-submit. Throughout this section, I'll use port 4000 as the address. For example, if you want to debug the driver program, you can add the following to your spark-submit command:

--driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=4000

After that, you should set your remote debugger to connect to the node where you have submitted the driver program. For the preceding case, port number 4000 was ...