Frank Kane's Taming Big Data with Apache Spark and Python

Counting word occurrences using flatmap()

We'll do a really common Spark and MapReduce example of dealing with a book or text file. We'll count all the words in a text file and find out how many times each word occurs within that text. We'll put a little bit of twist on this task and work our way up to doing more and more complex twists later on. The first thing we need to do is go over the difference again between map and flatMap, because using flatMap and Spark is going to be the key to doing this quickly and easily. Let's talk about that and then jump into some code later on and see it in action.