Frank Kane's Taming Big Data with Apache Spark and Python

Writing code to convert Marvel-Graph.txt to BFS nodes

A big part of the code is just the conversion of our source data into something that looks like a node. Then what we're going to do is convert that to an actual key/value pair-a key of a hero ID and a value that's a composite value of the list of connections, the distance and the color:

return (heroID, (connections, distance, color)) 

This way, we can group things together by hero ID more easily later.

Let's look at what is going on in the rest of the code. It's pretty straightforward stuff:

 def convertToBFS(line): fields = line.split() heroID = int(fields[0]) connections = [] for connection in fields[1:]: connections.append(int(connection)) color = 'WHITE' distance = 9999 if (heroID ...