Mastering Machine Learning with Spark 2.x

Term Frequency - Inverse Document Frequency (TF-IDF) weighting scheme

We will now use Spark ML to apply a very common weighting scheme called a TF-IDF to convert our tokenized reviews into vectors, which will be inputs to our machine learning models. The math behind this transformation is relatively straightforward:

For each token:

1. Find the term frequency within a given document (in our case, a movie review).
2. Multiply this count by the log of the inverse document frequency that looks at how common the token occurs among all of the documents (commonly referred to as the corpus).
3. Taking the inverse is useful, in that it will penalize tokens ...