Feature Engineering Made Easy

CountVectorizer

CountVectorizer is the most commonly used method to convert text data into their vector representations. It is similar to dummy variables, in the sense that CountVectorizer converts text columns into matrices where columns are tokens and cell values are counts of occurrences of each token in each document. The resulting matrix is referred to as a document-term matrix because each row will represent a document (in this case, a tweet) and each column represents a term (a word).

Let's take a look at a new dataset, and see how CountVectorizer works. The Twitter Sentiment Analysis dataset contains 1,578,627 classified tweets, and each row is marked as one for positive sentiment and zero for negative sentiment.

Further information ...