Modern Scala Projects

Step 1 – creating an RDD and preprocessing the data

Create a Scala file called BreastCancerRfPipeline.scala in the com.packt.modern.chapter2 package. Up until now, we relied on SparkSession and SparkContext, which are what spark-shell gave us. We need to create our SparkSession now, which will give us SparkContext. In BreastCancerRfPipeline.scala, after the package statement, place the following import statements:

import org.apache.spark.sql.SparkSession

 Create a SparkSession inside a trait, which we shall call WisconsinWrapper:

lazy val session: SparkSession = { SparkSession .builder() .master("local") .appName("breast-cancer-pipeline") .getOrCreate()

Just one SparkSession is made available to all classes extending from WisconsinWrapper ...