Modern Scala Projects

Step 13 – feature extraction

In this step, we will extract the features of this dataset. We will do the following:

• Creating feature vectors
• Creating features entails converting the text into bigrams of characters using an n-gram model
• These bigrams of characters will be hashed to a length 10000 feature vector 
• The final feature vector will be passed into Spark ML

Check out the following code snippet:

import org.apache.spark.ml.feature.HashingTFval hashMapper = new HashingTF().setInputCol("words").setOutputCol("noStopWordsMailFeatures").setOutputCol("mailFeatureHashes").setNumFeatures(10000)hashFeatures: org.apache.spark.ml.feature.HashingTF = hashingTF_5ff221eac4b4

Next, we will transform the featured version of the noStopWordsDataFrame ...