Scala and Spark for Big Data Analytics

combineByKey

combineByKey is very similar to aggregateByKey; in fact, combineByKey internally invokes combineByKeyWithClassTag, which is also invoked by aggregateByKey. As in aggregateByKey, the combineByKey also works by applying an operation within each partition and then between combiners.

combineByKey turns an RDD[K,V] into an RDD[K,C], where C is a list of Vs collected or combined under the name key K.

There are three functions expected when you call combineByKey.

• createCombiner, which turns a V into C, which is a one element list
• mergeValue to merge a V into a C by appending the V to the end of the list
• mergeCombiners to combine two Cs into one

In aggregateByKey, the first argument is simply a zero value but in combineByKey, we provide ...