Scala and Spark for Big Data Analytics

Submitting Spark jobs on YARN cluster

Now that our YARN cluster with the minimum requirement (for executing a small Spark job to be frank) is ready, to launch a Spark application in a cluster mode of YARN, you can use the following submit command:

$ SPARK_HOME/bin/spark-submit --classpath.to.your.Class --master yarn --deploy-mode cluster [options] <app jar> [app options]

For running our KMeansDemo, it should be done like this:

$ SPARK_HOME/bin/spark-submit      --class "com.chapter15.Clustering.KMeansDemo"      --master yarn      --deploy-mode cluster      --driver-memory 16g      --executor-memory 4g      --executor-cores 4      --queue the_queue      KMeans-0.0.1-SNAPSHOT-jar-with-dependencies.jar      Saratoga_NY_Homes.txt

The preceding submit command starts a YARN cluster ...