Scala and Spark for Big Data Analytics

reduce

reduce() applies the reduce function to all the elements in the RDD and sends it to the Driver.

The following is example code to illustrate this. You can use SparkContext and the parallelize function to create an RDD from a sequence of integers. Then you can add up all the numbers of the RDD using the reduce function on the RDD.

Since this is an action, the results are printed as soon as you run the reduce function.

Shown below is the code to build a simple RDD from a small array of numbers and then perform a reduce operation on the RDD:

scala> val rdd_one = sc.parallelize(Seq(1,2,3,4,5,6))rdd_one: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[26] at parallelize at <console>:24scala> rdd_one.take(10)res28: Array[Int] = Array(1, ...