Scala and Spark for Big Data Analytics

Actions and Transformations

RDDs are immutable and every operation creates a new RDD. Now, the two main operations that you can perform on an RDD are Transformations and Actions.

Transformations change the elements in the RDD such as splitting the input element, filtering out elements, and performing calculations of some sort. Several transformations can be performed in a sequence; however no execution takes place during the planning.

For transformations, Spark adds them to a DAG of computation and, only when driver requests some data, does this DAG actually gets executed. This is called lazy evaluation.

The reasoning behind the lazy evaluation is that Spark can look at all the transformations and plan the execution, making use of the understanding ...