Learning Data Mining with Python - Second Edition

Extracting word counts

We are going to use spaCy to extract our word counts. We still want to use it in a pipeline, but spaCy doesn't conform to our transformer interface. We will need to create a basic transformer to do this to obtain both fit and transform methods, enabling us to use this in a pipeline.

First, set up the transformer class. We don't need to fit anything in this class, as this transformer simply extracts the words in the document. Therefore, our fit is an empty function, except that it returns self which is necessary for transformer objects to conform to the scikit-learn API.

Our transform is a little more complicated. We want to extract each word from each document and record True if it was discovered. We are only using ...