High Performance Spark, 2nd Edition

Appendix D. Advanced Task Scheduling: Gangs and Resource Profiles

Spark’s default scheduler works well in cases where individual tasks can be retried and resources are, if not equal, fairly similar per executor. You can use resource profiles when specific tasks require unique resources, such as more memory or hardware accelerators (e.g., GPUs). Similarly, Gang Scheduling, also called barrier execution mode in Spark, solves the problem of requiring all tasks in a given stage to execute at the same time and of retrying an entire block of computation, not just a single partition, on failure. Gang scheduling is most often used for specific ML workloads.

You designate certain operations as needing gang scheduling/barrier execution mode or specific resource profiles on RDDs and Datasets/DataFrames.

Gang Scheduling/Barrier Execution Mode

On RDDs, for gang scheduling you just need to call barrier, which will give you back a special BarrierRDD. The operations you do on the BarrierRDD will be gang/barrier scheduled, and you can use mapPartitions to transform each partition.

On Datasets/DataFrames the API assumes you are using barrier execution mode in Python and as such is available for mapInPandas and mapInArrow as a boolean flag.

Resource Profiles

Resource profiles have a similar RDD and Dataset/DataFrame API but require a bit of extra setup work to predefine the different resource profiles. On RDDs you call the withResources function, and on DataFrames you pass in the ResourceProfile ...