Frank Kane's Taming Big Data with Apache Spark and Python

How do we know when we're done?

We'll just go through this process over and over again until we encounter the character we're interested in. At that point, we can count how many iterations we had to go through to figure out the degrees of separation. But how do we know when we're done? I mean, this is running potentially on the cluster, and there's no set way of knowing when we've actually encountered the character we care about. Well, that's where accumulators come in.

Remember how broadcast variables let you share objects across all the nodes in your cluster? Well, an accumulator allows all the executors in your cluster to increment some shared variable. It's basically a counter that is maintained and synchronized across all of the different ...