Frank Kane's Taming Big Data with Apache Spark and Python

Strategy

Let's outline our strategy for finding the most popular superhero, or more specifically, the one that had the most appearances with other superheroes or the most co-occurrences:

• Map input data to hero ID and number of co-occurrences per line:
  • We're going to start off by parsing out our file and map that input data to a key/value pair RDD of hero ID to number of co-occurrences per line. We can count up how many other heroes they appeared with per line.
• Add up co-occurrence by hero ID using reduceBYKey():
  • Since we can actually have heroes that span multiple lines, we're going to need to add them up using reduceByKey. This'll give us the final count of how many occurrences by hero ID exist.
• Flip the (map) RDD to (number, hero ...