Machine Learning Algorithms - Second Edition

Stopword removal

Stopwords are part of normal speech (articles, conjunctions, and so on), but their occurrence frequency is very high and they don't provide any useful semantic information. For these reasons, it's a good practice to filter sentences and corpora by removing them all. NLTK provides lists of stopwords for the most common languages, and their usage is immediate:

from nltk.corpus import stopwordssw = set(stopwords.words('english'))

A subset of English stopwords is shown in the following snippet:

print(sw){u'a', u'about', u'above', u'after', u'again', u'against', u'ain'...

To filter a sentence, it's possible to adopt a functional approach:

complex_text = 'This isn\'t a simple text. Count 1, 2, 3 and then go!'ret = RegexpTokenizer('[a-zA-Z\']+') ...