Practical Real-time Data Processing and Analytics

Shared variables – broadcast variables and accumulators

While working in distributed compute programs and modules, where the code executes on different nodes and/or different workers, a lot of time a need arises to share data across the execution units in the distributed execution setup. Thus Spark has the concept of shared variables. The shared variables are used to share information between the parallel executing tasks across various workers or the tasks and the drivers. Spark supports two types of shared variable:

• Broadcast variables
• Accumulators

In the following sections, we will look at these two types of Spark variables, both conceptually and pragmatically.