Machine Learning for OpenCV 4 - Second Edition

Using TF-IDF to improve the result

It was called the Term Frequency-Inverse Document Frequency (TF-IDF), and we encountered it in Chapter 4, Representing Data and Engineering Features. If you recall, what TF-IDF does is basically weigh the word count by a measure of how often the words appear in the entire dataset. A useful side effect of this method is the IDF part—the inverse frequency of words. This makes sure that frequent words, such as and, the, and but, carry only a small weight in the classification.

We apply TF-IDF to the feature matrix by calling fit_transform on our existing feature matrix, X:

In [24]: tfidf = feature_extraction.text.TfidfTransformer()In [25]: X_new = tfidf.fit_transform(X)

Don't forget to split the data; also, ...