Scala and Spark for Big Data Analytics

Submitting Spark jobs

Once a Spark application is bundled as either a jar file (written in Scala or Java) or a Python file, it can be submitted using the Spark-submit script located under the bin directory in Spark distribution (aka $SPARK_HOME/bin). According to the API documentation provided in Spark website (http://spark.apache.org/docs/latest/submitting-applications.html), the script takes care of the following:

• Setting up the classpath of JAVA_HOME, SCALA_HOME with Spark
• Setting up the all the dependencies required to execute the jobs
• Managing different cluster managers
• Finally, deploying models that Spark supports

In a nutshell, Spark job submission syntax is as follows:

$ spark-submit [options] <app-jar | python-file> [app arguments] ...