Feature Engineering Made Easy

Using text in machine learning pipelines

Of course, the ultimate goal of our vectorizers is to use them to make text data ingestible for our machine learning pipelines. Because CountVectorizer and TfidfVectorizer act like any other transformer we have been working with in this book, we will have to utilize a scikit-learn pipeline to ensure accuracy and honesty in our machine learning pipeline. In our example, we are going to be working with a large number of columns (in the hundreds of thousands), so I will use a classifier that is known to be more efficient in this case, a Naive Bayes model:

from sklearn.naive_bayes import MultinomialNB # for faster predictions with large number of features...

Before we start building our pipelines, let's ...