Frank Kane's Taming Big Data with Apache Spark and Python

Text normalization

In the previous section, we had a first crack at counting the number of times each word occurred in our book, but the results weren't that great. We had each individual word that had different capitalization or punctuation surrounding it being counted as a word of its own, and that's not what we want. We want each word to be counted only once, no matter how it's capitalized or what punctuation might surround it. We don't want duplicate words showing up in there. There are toolkits you can get for Python such as NLTK (Natural Language Toolkit), that have fancy, very complicated ways of normalizing text data like this and automatically figuring out what words are the same. However, we can sort of do a more simplistic attack ...