Frank Kane's Taming Big Data with Apache Spark and Python

Transforming RDDs

One common thing you're going to do once you have an RDD is transform it in some way, shape, or form. The following list shows some of the basic operations you can do on RDDs. This is not a complete list, but these are all the most common operations you can do on an RDD:

• map
• flatmap
• filter
• distinct
• sample
• union, intersection, subtract, cartesian

There's not a whole lot to wrap your head around here. The thing is, although there aren't a lot of different operations you can do to transform an RDD, they're all very powerful. So let's start with the map function on an RDD. This allows you to take a set of data and transform it into some other set of data, given a function that operates on the RDD. So, for example, if I want ...