Operations in Spark

RDDs support two types of operations:

  • Transformations
  • Actions


The transformation operation performs some functions and creates another dataset. Transformations are processed in the lazy mode and only those transformations that are needed in the end result are processed. If any transformation is found unnecessary, then Spark ignores it, and this improves the efficiency.

Transformations, which are available and mentioned in Spark Apache docs at https://spark.apache.org/docs/latest/programming-guide.html#transformations, are as follows:



map (func)

Return a new distributed dataset formed by passing each element of the source through a function func.

filter (func)

Return a new dataset formed ...

Get Hadoop Essentials now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.