Apache Spark Deep Learning Cookbook

There's more...

There were a couple of column manipulations done with the pyspark.sql module in this section. The withColumn() operator returns a new dataframe or modifies an existing dataframe by adding a new column or modifies an existing column of the same name. This is not to be confused with the withColumnRenamed() operator, which also returns a new dataframe, but by modifying the name of an existing column to a new column. Finally, we needed to perform some logical operations to convert values associated with Fire to 0 and without Fire to 1. This required using the pyspark.sql.functions module and incorporating the where function as an equivalent to a case statement used in SQL. The function created a case statement equation using the ...