Frank Kane's Taming Big Data with Apache Spark and Python

Convert the input file into structured data

The first thing we need to do is actually convert our data file or input file into something that looks like the nodes that we described in the BFS algorithm in the previous section, Superhero degrees of separation - introducing breadth-first search.

We're starting off, for example, with a line of input that looks like the one shown here that says hero ID 5983 appeared with heroes 1165, 3836, 4361, and 1282:

We want to convert that to something a little bit more interesting and useful. We want to convert that to structured data, where we have for example, the hero ID in question, the list of their ...