March 2017
Beginner to intermediate
356 pages
7h 11m
English
As you have seen in the previous example, passing functions is a critical functionality provided by Spark. From a user's point of view you would pass the function in your driver program, and Spark would figure out the location of the data partitions across the cluster memory, running it in parallel. The exact syntax of passing functions differs by the programming language. Since Spark has been written in Scala, we'll discuss Scala first.
In Scala, the recommended ways to pass functions to the Spark framework are as follows:
Anonymous functions are used for short pieces of code. They are also referred to as lambda expressions, and are a cool and elegant ...
Read now
Unlock full access