Frank Kane's Taming Big Data with Apache Spark and Python

Calling flatMap()

Okay, what happens next? Next, down in line 103, we're going to go through some upper bound of iteration. So we're going to assume that we're never going to be more than ten degrees of separation from anybody in this graph, if so, they're probably not connected at all. In this line, we're just picking some arbitrary upper bound, here, 10. We will iterate through the graph, doing the BFS traversal up to ten times. The first time we'll say we're running the BFS iteration number 10:

for iteration in range(0, 10): 
    print("Running BFS iteration# " + str(iteration+1)) 

Then we will call our flatMap. This runs that first step, as we talked about in the previous section, of blowing out all the gray nodes. This is calling our bfsMap ...