Apache Spark Quick Start Guide

Creating UDFs

Users can define User-Defined Functions (UDFs) for custom logic in Scala or Python. The formats for UDF definition and registration are explained as follows:

•  The syntax for registering a Spark SQL function as a UDF in Scala is given as follows:

val squared = (s: Int) => { s * s }
spark.udf.register("square", squared)

•  The syntax for calling a Spark SQL function as a UDF in Scala is given as follows:

spark.range(1, 20).createOrReplaceTempView(("udf_test"))%sql select id, square(id) as id_squared from udf_test

•  The syntax for registering a Spark SQL function as a UDF in Python is given as follows:

def squared(s):
  return s * s
spark.udf.register("squaredWithPython", squared)

•  The syntax for calling a Spark SQL function as ...