Frank Kane's Taming Big Data with Apache Spark and Python

Filter out all but the TMIN entries

Alright, now we're going to call filter on our RDD. Now you remember that we stored the entry type in field number 1, we start counting at 0, so field 0 is the station ID, 1 is the entry type, and 2 is the temperature. If we see TMIN in field number 1 of our value, we're going to let that pass, otherwise we will discard it. So we're going to end up with here is a new RDD called minTemps that contains only TMIN entry type observations:

minTemps = parsedLines.filter(lambda x: "TMIN" in x[1]) 

The actual format of the RDD will be the same, we're still going to have station ID, entry type, and temperature-composite values, lists, if you will-but we're only going to have entry types that contain TMIN at this ...