March 2017
Beginner to intermediate
356 pages
7h 11m
English
Python provides a simple way to pass functions to Spark. The Spark programming guide available at spark.apache.org suggests there are three recommended ways to do this:
defs inside the function calling into Spark for longer codeWhile we have already looked at the lambda functions in some of the previous examples, let's look at local definitions of the functions. We can encapsulate our business logic which is splitting of words, and counting into two separate functions as shown below.
def splitter(lineOfText): words = lineOfText.split(" ") return len(words) def aggregate(numWordsLine1, ...Read now
Unlock full access