Scala and Spark for Big Data Analytics

Level of parallelism

Although you can control the number of map tasks to be executed through optional parameters to the SparkContext.text file, Spark sets the same on each file according to its size automatically. In addition to this, for a distributed reduce operation such as groupByKey and reduceByKey, Spark uses the largest parent RDD's number of partitions. However, sometimes, we make one mistake, that is, not utilizing the full computing resources for your nodes in a computing cluster. As a result, the full computing resources will not be fully exploited unless you set and specify the level of parallelism for your Spark job explicitly. Therefore, you should set the level of parallelism as the second argument.

For more on this option, ...