Scala and Spark for Big Data Analytics

flatMap function

flatMap() applies transformation function to input partitions to generate output partitions in the output RDD just like map() function. However, flatMap() also flattens any collection in the input RDD elements.

As shown in the following snippet, we can use flatMap() on a RDD of a text file to convert the lines in the text to a RDD containing the individual words. We also show map() called on the same RDD before flatMap() is called just to show the difference in behavior:

scala> val rdd_two = sc.textFile("wiki1.txt")rdd_two: org.apache.spark.rdd.RDD[String] = wiki1.txt MapPartitionsRDD[8] at textFile at <console>:24scala> rdd_two.countres6: Long = 9scala> rdd_two.firstres7: String = Apache Spark provides programmers with an ...