Machine Learning for OpenCV 4 - Second Edition

Preprocessing the data

Scikit-learn offers several options when it comes to encoding text features, which we discussed in Chapter 4, Representing Data and Engineering Features. One of the simplest methods of encoding text data, as you may recall, is by word count; for each phrase, you count the number of occurrences of each word within it. In scikit-learn, this is easily done using CountVectorizer:

In [10]: from sklearn import feature_extraction...      counts = feature_extraction.text.CountVectorizer()...      X = counts.fit_transform(data['text'].values)...      X.shapeOut[10]: (52076, 643270)

The result is a giant matrix, which tells us that we harvested a total of 52,076 emails that collectively contain 643,270 different words. However, scikit-learn ...