Operations in Spark

RDDs support two types of operations:

  • Transformations
  • Actions

Transformations

The transformation operation performs some functions and creates another dataset. Transformations are processed in the lazy mode and only those transformations that are needed in the end result are processed. If any transformation is found unnecessary, then Spark ignores it, and this improves the efficiency.

Transformations, which are available and mentioned in Spark Apache docs at https://spark.apache.org/docs/latest/programming-guide.html#transformations, are as follows:

Transformation

Meaning

map (func)

Return a new distributed dataset formed by passing each element of the source through a function func.

filter (func)

Return a new dataset formed ...

Get Hadoop Essentials now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.