Although broadcast and accumulator variables are simple and very limited variables (one is read-only, and the other one is write only), they can be actively used to create very complex operations. For example, let's try to apply different machine learning algorithms on the iris dataset in a distributed environment. We will build a Spark job in the following way:
- The dataset is read and broadcast to all the nodes (as it's small enough to fit in-memory).
- Each node will use a different classifier on the dataset and return the classifier name and its accuracy score on the full dataset. Note that, to keep things easy in this simple example, we won't do any preprocessing, train/test splitting, ...
