Building Machine Learning Systems with Python - Third Edition

Our achievements and goals

Our current text preprocessing phase includes the following steps:

1. Firstly, tokenizing the text
2. This is followed by throwing away words that occur way too often to be of any help in detecting relevant posts
3. Throwing away words that occur so infrequently that there is little chance that they will occur in future posts
4. Counting the remaining words
5. Finally, calculating TF-IDF values from the counts, considering the whole text corpus

Again, we can congratulate ourselves. With this process, we are able to convert a bunch of noisy text into a concise representation of feature values.

But as simple and powerful the bag of words approach with its extensions is, it has some drawbacks, which we should be aware of:

• It does ...