Frank Kane's Taming Big Data with Apache Spark and Python

Strategy for solving the problem

So what should your general strategy be? I'll help you think through how to attack this problem. The first thing you want to do is write a map function that will read the input data and split it up based on comma-delimited fields. We've done that before, so go back and look at some of the other examples in your SparkCourse folder for a reminder on how that works. After doing that, you want to map each line to key/value pairs of customer IDs and amount spent per transaction. As we're going to structure this as a key/value RDD that allows us to then use reduceByKey to sum up all the amounts spent by customer ID. Finally, all you have to do is call collect() on the resulting RDD that reduceByKey gives you and ...