Stemming is the action of reducing inflectional forms of words and taking the words to their core concepts. For example, the concept behind is, be, are, and am is the same. Similarly, the concept behind go and goes, as well as table and tables, is the same. The operation of deriving the root concept for each word is called stemming. In NLTK, you can choose the stemmer that you'd like to use (there are several ways to get the root part of words). We'll show you one of them, letting the others in Jupyter Notebook associated with this part of the book:
In: from nltk.stem import * stemmer = LancasterStemmer() print ([stemmer.stem(word) for word in nltk_tokens])Out: ['the', 'coolest', 'job', 'in', 'the', 'next', '10', 'year', 'wil', ...
