March 2017
Beginner to intermediate
356 pages
7h 11m
English
We have seen how easy it was to launch a cluster with minimal effort. But, the cluster that we had set up was a relatively smallish cluster with only 5 worker nodes. Imagine, setting up a 5000 nodes cluster following the above steps. This will neither be easy nor maintainable especially in terms of adding/removing nodes from this cluster. Spark therefore provides options by which you can make sure you don't have to manually perform such configuration.
Before running any scripts you have to make sure that the workers are accessible from the master via SSH. You will need to provide either of the following:
SPARK_SSH_FOREGROUND ...Read now
Unlock full access