Apache Spark Quick Start Guide

Choosing the right transformations

One of the ways to optimize your applications is to avoid the shuffling of your data. Lesser the shuffle, less in the execution time. If you choose your transformations carefully, then you can avoid heavy data shuffles. For example, let's assume you have an RDD of tuples (pair RDD) having the first element as an alphabet and the second element a numeric value. You want to compute the sum of all the elements of the RDD based on the keys. There are many ways to do that but, let's look at two of the choices:

• groupByKey()

//groupByKey()val rawData = Array(("A",1),("B",1),("C",1),("A",2),("B",1))val baseRDD = spark.sparkContext.parallelize(rawData)baseRDD.groupByKey().mapValues(_.sum).collect()//Outputres20: ...