Machine Learning Algorithms - Second Edition

Count vectorizing

The algorithm is very simple and it's based on representing a token considering how many times it appears in a document. Of course, the whole corpus must be processed to determine how many unique tokens are present and their frequencies. Let's see an example of the CountVectorizer class on a simple corpus:

from sklearn.feature_extraction.text import CountVectorizercorpus = [       'This is a simple test corpus',       'A corpus is a set of text documents',       'We want to analyze the corpus and the documents',       'Documents can be automatically tokenized']cv = CountVectorizer()vectorized_corpus = cv.fit_transform(corpus)print(vectorized_corpus.todense())[[0 0 0 0 0 1 0 1 0 0 1 1 0 0 1 0 0 0 0] [0 0 0 0 0 1 1 1 1 1 0 0 1 0 0 0 0 0 0] [1 1 0 ...