Using text in machine learning pipelines

Of course, the ultimate goal of our vectorizers is to use them to make text data ingestible for our machine learning pipelines. Because CountVectorizer and TfidfVectorizer act like any other transformer we have been working with in this book, we will have to utilize a scikit-learn pipeline to ensure accuracy and honesty in our machine learning pipeline. In our example, we are going to be working with a large number of columns (in the hundreds of thousands), so I will use a classifier that is known to be more efficient in this case, a Naive Bayes model:

from sklearn.naive_bayes import MultinomialNB # for faster predictions with large number of features...

Before we start building our pipelines, let's ...

Get Feature Engineering Made Easy now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.