Apache Spark 2.x Cookbook

Garbage collection

To understand memory optimization, it is a good idea to understand how memory management works in Java. Objects reside in heap in Java. Heap is created when JVM starts, and it can resize itself when needed (based on the minimum and maximum size, that is, -Xms and -Xmx, respectively, assigned in the configuration).

Heap is divided into two spaces or generations: young space and old space, as shown in the following figure:

The young space is reserved for the allocation of new objects. The young space consists of an area called Eden and two smaller survivor spaces. When the young space becomes full, garbage is collected by ...