Apache Spark Quick Start Guide

Broadcast variables

Before we explain the broadcast variable, let's discuss the concept of closures. Let's take a simple example:

#Python spark.sparkContext.parallelize(range(1,11))\                  .map(lambda x : x*2)\                  .collect()#OutputOut[4]: [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]

In the previous example, we first created an RDD by using parallelize() method and then we multiplied each element by 2 to generate the even numbers. Under the hood, when Spark driver schedules the tasks, each task has a copy of the function (lambda x: x*2) along with the data that function uses (in our example: 2 ). This means each task will have a copy of the number 2. In our example, you might not notice any performance bottleneck as our data size is very small and also we ...