Hands-On Recommendation Systems with Python

TF-IDFVectorizer

Not all words in a document carry equal weight. We already observed this when we eliminated the stop words from our vocabulary altogether. But the words that were in the vocabulary were all given equal weighting.

But should this always be the case?

For example, consider a corpus of documents on dogs. Now, it is obvious that all these documents will frequently contain the word dog. Therefore, the appearance of the word dog isn't as important as another word that only appears in a few documents.

TF-IDFVectorizer (Term Frequency-Inverse Document Frequency)takes the aforementioned point into consideration and assigns weights to each word according to the following formula. For every word i in document j, the following applies: ...