Mastering Machine Learning with Spark 2.x

Preparation of the environment

In this Chapter, instead of using Spark shell, we will build two standalone Spark applications using Scala API: one for model preparation and the second for model deployment. In the case of Spark, the Spark application is a normal Scala application with a main method that serves as an entry point for execution. For example, here is a skeleton of application for model training:

object Chapter8 extends App {val spark = SparkSession.builder()     .master("local[*]")     .appName("Chapter8")     .getOrCreate()val sc = spark.sparkContextsc.setLogLevel("WARN")script(spark, sc, spark.sqlContext)def script(spark: SparkSession, sc: SparkContext, sqlContext: SQLContext): Unit = {      // ...code of application
}
}
  

Moreover, we will try ...