Frank Kane's Taming Big Data with Apache Spark and Python

Degrees of separation

What we're going to be doing is play with superheroes and our superhero dataset. We're going to talk about an algorithm called breadth-first search, and use it to find the degrees of separation between superheroes. Along the way, we'll introduce a concept called accumulators. Accumulators, kind of like different executors, keep track of something together. Let's talk about how we're going to go about this. Remember, what we're trying to do is figure out the degrees of separation between superheroes. We want to find out which character is the Kevin Bacon of superheroes.

Look at the following figure. Let's say, we want to find out how far apart Iron Man is from Spider-Man, and these lines all indicate which superheroes ...