Frank Kane's Taming Big Data with Apache Spark and Python

Input data format

Let's take a look at our input data. Basically, there are two files that we're going to include. One is called Marvel-graph.txt and is something I prepared ahead of time. The format of this data is a given superhero ID, which in the case shown as follows is 4395, followed by a list of all the superhero IDs that character 4395 appeared with in other comic books:

So, every line has the first entry as a given hero, followed by list of all the other heroes that they appeared with, and so on and so forth. It's important to note that a hero may span multiple lines, so in our source data they would actually break really long lines ...