Frank Kane's Taming Big Data with Apache Spark and Python

Extract (MAP) the data we care about

This image demonstrates what our mapper function does:

The Lambda sequence allows us to have a shorthand of passing a function that we want to pass into our map. So you can see, we have lines.map and then inside our parentheses, we have lambda x, where x gets passed into split, and we extract the 2 number field:

ratings = lines.map(lambda x: x.split()[2]) 

What this code is going to do, in every line, is take the line of input and split it based on whitespace individual fields. Running split on the first line, for example, is going to result in a list of values of 196 242 3 881250949. These numbers represent ...