Apache Spark 2.x Cookbook

How to do it...

1. Load Spark shell or Databricks Cloud:

        $ spark-shell

2. Do the necessary imports:

        scala> import org.apache.spark.ml.feature._ 

3. Load the Charles Dickens novel Hard Times:

        scala> val sentenceData =         spark.read.option("delimiter","\n").csv        ("s3a://sparkcookbook/hardtimes").toDF("sentence")

4. Create a transformer to tokenize sentences:

        scala> val tokenizer = new         Tokenizer().setInputCol("sentence").setOutputCol("words")

5. Tokenize sentences:

        scala> val wordsData = tokenizer.transform(sentenceData)

6. Create a hashing transformer:

        scala> val hashingTF = new HashingTF()        .setInputCol("words").setOutputCol("rawFeatures").setNumFeatures(20)

7. Map the sequence of terms to term frequencies:

 scala> val featurizedData ...