Frank Kane's Taming Big Data with Apache Spark and Python

Mapping input data to (hero ID, number of co-occurrences) per line

The first thing we do, if you look at line 14, is load up our names.txt file into an RDD called names using sc.textFile:

names = sc.textFile("file:///SparkCourse/marvel-names.txt") 

We're going to do this name lookup a little bit differently than we did in the previous example. Instead of using a broadcast variable, we'll store the superhero ID to name information as a key/value RDD. This too can be used across different nodes on your cluster if necessary. This is just another way to solve the same problem.

After we load up that text file, we use the parseNames function to map it from lines of input data to actual key/value pairs:

namesRdd = names.map(parseNames) 

If we ...