Natural Language Processing and Computational Linguistics

Starting clustering

Like every other text analysis algorithm we applied before, the most important step remains the preprocessing step — getting rid of our stop words and lemmatizing words.

Once we're done with this, the next step is to convert our document into a vector representation we are most comfortable with.

Since we're dealing with scikit-learn's implementations for clustering and classification, let us use scikit-learn for our preprocessing. We should also use this opportunity to decide which dataset we intend to use for our experiments. While there are lots of solid options, we will stick with the popular 20 Newsgroups [3] dataset. Since the dataset comes bundled with scikit-learn, loading it and using it becomes an easy task as ...