Apache Spark Quick Start Guide

Parallelism 

The default number of shuffle partitions in Spark SQL is 200. It means if you use a transformation that shuffles the data, you would have 200 shuffle blocks as an output. If your data is huge and you still use the default setting of spark.sql.shuffle.partitions then the shuffle blocks will have a high block size. In Spark, the size of the shuffle block cannot be greater than 2 GB. This might lead to a run time exception. To avoid such exceptions, you should always consider increasing the number of shuffle partitions. This will eventually decrease the size of the shuffle partitions. The ideal size of a partition is close to 128 MB. In Spark SQL, you can change the size of the shuffle blocks by changing the value of spark.sql.shuffle.partitions ...