Frank Kane's Taming Big Data with Apache Spark and Python

Find minimum temperature by station ID

Given that, we want to find the minimum temperature for the entire year for every station, we can call reduceByKey to do that. The reduceByKey function is going to aggregate together every minimum temperature observed for every weather station ID, and then our lambda function will determine how we do that aggregation:

minTemps = stationTemps.reduceByKey(lambda x, y: min(x,y)) 

In that lambda function, we're saying that for whenever we try to combine two observations together for a minimum temperature for a given station, we're going to call the min function to actually only take the minimum value between those two. As we keep feeding more and more observations for each weather station in there, only ...