Scala and Spark for Big Data Analytics

Debugging Spark application using SBT

The preceding setting works mostly on Eclipse or IntelliJ using the Maven project. Suppose that you already have your application done and are working on your preferred IDEs such as IntelliJ or Eclipse as follows:

object DebugTestSBT {  def main(args: Array[String]): Unit = {    val spark = SparkSession      .builder      .master("local[*]")      .config("spark.sql.warehouse.dir", "C:/Exp/")      .appName("Logging")      .getOrCreate()          spark.sparkContext.setCheckpointDir("C:/Exp/")    println("-------------Attach debugger now!--------------")    Thread.sleep(8000)    // code goes here, with breakpoints set on the lines you want to pause  }}

Now, if you want to get this job to the local cluster (standalone), the very first step is packaging ...