Scala and Spark for Big Data Analytics

HashPartitioner

HashPartitioner is the default partitioner in Spark and works by calculating a hash value for each key of the RDD elements. All the elements with the same hashcode end up in the same partition as shown in the following code snippet:

partitionIndex = hashcode(key) % numPartitions

The following is an example of the String hashCode() function and how we can generate partitionIndex:

scala> val str = "hello"str: String = helloscala> str.hashCoderes206: Int = 99162322scala> val numPartitions = 8numPartitions: Int = 8scala> val partitionIndex = str.hashCode % numPartitionspartitionIndex: Int = 2

The default number of partitions is either from the Spark configuration parameter spark.default.parallelism or the number of cores in the ...