Scala and Spark for Big Data Analytics

Advance job submissions in a YARN cluster

If you opt for the more advanced way of submitting Spark jobs to be computed in your YARN cluster, you can specify additional parameters. For example, if you want to enable the dynamic resource allocation, make the spark.dynamicAllocation.enabled parameter true. However, to do so, you also need to specify minExecutors, maxExecutors, and initialExecutors as explained in the following. On the other hand, if you want to enable the shuffling service, set spark.shuffle.service.enabled as true. Finally, you could also try specifying how many executor instances will be running using the spark.executor.instances parameter.

Now, to make the preceding discussion more concrete, you can refer to the following ...