Frank Kane's Taming Big Data with Apache Spark and Python

What is filter()

Filter is just another function you can call on a mapper, which transforms it by removing information that you don't care about. In our example, the raw weather data actually includes things such as minimum temperatures observed and maximum temperatures for every day, and also the amount of precipitation observed for every day. However, all we care about for the problem we're trying to solve is the minimum temperature observed at a given station for the year. So we need to apply a filter that just looks for weather data types of TMIN and allows those to pass through our new RDD that we're calling minTemps as you can see in this line of code:

minTemps = parsedLines.filter(lambda x: "TMIN" in x[1]) 

Everything else like "TMAX" ...