Python Machine Learning By Example

Data preprocessing

We see items, which are obviously not words, such as 00 and 000. Maybe we should ignore items that contain only digits. However, 0d and 0t are also not words. We also see items as __, so maybe we should only allow items that consist only of letters. The posts contain names such as andrew as well. We can filter names with the Names corpus from NLTK we just worked with. Of course, with every filtering we apply, we have to make sure that we don't lose information. Finally, we see words that are very similar, such as try and trying, and word and words.

We have two basic strategies to deal words from the same root--stemming and lemmatization. Stemming is the more quick and dirty type approach. It involves chopping, if necessary, ...