Java Deep Learning Cookbook

Memory settings and garbage collection for Spark

Memory management is very crucial for distributed training with large datasets in production. It directly influences the resource consumption and performance of the neural network. Memory management involves configuring off-heap and on-heap memory spaces. DL4J/ND4J-specific memory configuration will be discussed in detail in Chapter 12, Benchmarking and Neural Network Optimization.

In this recipe, we will focus on memory configuration in the context of Spark.