Scala and Spark for Big Data Analytics

HashingTF

HashingTF is a Transformer, which takes a set of terms and converts them into vectors of fixed length by hashing each term using a hash function to generate an index for each term. Then, term frequencies are generated using the indices of the hash table.

In Spark, the HashingTF uses the MurmurHash3 algorithm to hash terms.

In order to use HashingTF, you need to import the following package:

import org.apache.spark.ml.feature.HashingTF

First, you need to initialize a HashingTF specifying the input column and the output column. Here, we choose the filtered words column created by the StopWordsRemover Transformer and generate an output column rawFeaturesDF. We also choose the number of features as 100:

scala> val hashingTF = new HashingTF().setInputCol("filteredWords").setOutputCol("rawFeatures").setNumFeatures(100) ...