Feature Engineering Made Easy

CountVectorizer parameters

A few parameters that we will go over include:

• stop_words
• min_df
• max_df
• ngram_range
• analyzer

stop_words is a frequently used parameter in CountVectorizer. You can pass in the string english to this parameter, and a built-in stop word list for English is used. You can also specify a list of words yourself. These words will then be removed from the tokens and will not appear as features in your data.

Here is an example:

vect = CountVectorizer(stop_words='english')  # removes a set of english stop words (if, a, the, etc)
_ = vect.fit_transform(X)
print _.shape(99989, 105545)

You can see that the feature columns have gone down from 105,849 when stop words were not used, to 105,545 when English stop words have ...