Java Deep Learning Cookbook

How it works...

In step 1, we discussed Spark-specific memory configurations. We mentioned that this can be configured for master/worker nodes. Also, these memory configurations can be dependent on the cluster resource manager.

Note that the --executor-memory 4g command-line argument is for YARN. Please refer to the respective cluster resource manager documentation to find out the respective command-line argument for the following:

• Spark Standalone: https://spark.apache.org/docs/latest/spark-standalone.html
• Mesos: https://spark.apache.org/docs/latest/running-on-mesos.html
• YARN: https://spark.apache.org/docs/latest/running-on-yarn.html

For Spark Standalone, use the following command-line options to configure the memory space:

• The on-heap ...