Scala and Spark for Big Data Analytics

groupByKey

groupByKey groups the values for each key in the RDD into a single sequence. groupByKey also allows controlling the partitioning of the resulting key-value pair RDD by passing a partitioner. By default, a HashPartitioner is used but a custom partitioner can be given as an argument. The ordering of elements within each group is not guaranteed, and may even differ each time the resulting RDD is evaluated.

groupByKey is an expensive operation due to all the data shuffling needed. reduceByKey or aggregateByKey provide much better performance. We will look at this later in this section.

groupByKey can be invoked either using a custom partitioner or just using the default HashPartitioner as shown in the following code snippet:

def groupByKey(partitioner: ...