Hands-On Data Science with Anaconda

Parallel processing for word frequency

First, let's look at a simple Python program to find out the first most frequently used word in an input text file. We randomly chose Da Vinci Code at http://www.gutenberg.org/files/5000/5000-8.txt. Assume that the downloaded novel is saved under c:/temp/daVinci.txt. The following Python code will list the top 10 most frequent words:

text = open("c:/temp/daVinci.txt",'r').read().lower() def byFreq(pair): return pair[1] for ch in '!"#$%&()*+,-./:;<=>?@[\]^_`{|}~': text = text.replace(ch, ' ') words = text.split() counts = {} for w in words: counts[w] = counts.get(w,0) + 1 n = 10 # for the first n most frequetly used words words= list(counts.items()) words.sort() words.sort(key=byFreq, reverse=True) for ...