Machine Learning with Spark - Second Edition

Building a tf-idf model

We will now use Spark ML to transform each document, in the form of processed tokens, into a vector representation. The first step will be to use the HashingTF implementation, which makes use of feature hashing to map each token in the input text to an index in the vector of term frequencies. Then, we will compute the global IDF and use it to transform the term frequency vectors into tf-idf vectors.

For each token, the index will thus be the hash of the token (mapped in turn onto the dimension of the feature vector). The value for each token will be the tf-idf weighting for that token (that is, the term frequency multiplied by the inverse document frequency).

First, we will import the classes we need and create our ...