Apache Spark Quick Start Guide

mapPartitions()

The mapPartitions() transformation is similar to map(). It also allows users to manipulate elements of an RDD, but it provides more control at a per-partition basis. It applies a function that accepts an iterator as an argument and returns an iterator as the output. If you have done some shell scripting and you are aware of AWK programming, then you can correlate that with mapPartitions transformation to understand it better. A typical AWK example looks something like BEGIN { #Begin block } { #middle block } END { #end Block }. The Begin block executes only once before reading the file content, the middle block executes for each line in the input file, and the end block also executes only once at the end of the file. Similarly, ...