Spark is a complex distributed computing framework, and has many moving parts. Various cluster resources, such as memory, CPU, and network bandwidth, can become bottlenecks at various points. As Spark is an in-memory compute framework, the impact of the memory is the biggest.
Another issue is that it is common for Spark applications to use a huge amount of memory, sometimes more than 100 GB. This amount of memory usage is not common in traditional Java applications.
In Spark, there are two places where memory optimization is needed, and that is at the driver and at the executor level.
You can use the following commands to set the driver memory:
$ spark-shell --drive-memory 4g
$ spark-submit --drive-memory ...