Frank Kane's Taming Big Data with Apache Spark and Python

User-defined functions (UDFs)

One last thing to talk about SparkSQL is that you can also have user-defined functions, so you can create your own functions. For example, in the following piece of code, we'll define a square function that takes x and returns x times x:

from pyspark sql.types import IntegerType 
hiveCtx.registerFunction("square", lambda x: x*x, IntegerType()) 
df = hiveCtx.sql("SELECT square('someNumericField') FROM tableName) 

Once you define this type of function and register it, you can use it within your SQL queries. So you can extend the SQL syntax yourselves to do specialized operations, which will get executed across your cluster automatically.

Let's look at a few real examples of using SparkSQL. We're going to do three ...