Frank Kane's Taming Big Data with Apache Spark and Python

Introducing broadcast variables

Spark offers something called as a broadcast object. The way it works is you take your SparkContext object and you call broadcast() on it as a parameter, you will then ship off whatever object you want to broadcast to every node:

sc.broadcast() 

This will do all the work of serializing it and sending it across the wire and making it available to every executor node in your cluster. Then from your code, you can just call .value() to actually retrieve the object from the broadcasted object. It'll make a little more sense when we look at an example, so let's go off to the code and actually make that happen.