Frank Kane's Taming Big Data with Apache Spark and Python

Examining the script

Usually, this is a really simple example in textbooks, but we're going to make it something a little bit more interesting. If you haven't already, go to the download package for this book, download word-count-better-sorted.py, and open that up in your favorite Python editor:

So what have we done differently here?

Instead of just cheating and calling countByValue, we're actually implementing countByValue by hand. We have a mapper where we transform every word into a key/value pair of the word and the number 1. Then we use ...