Hands-On Big Data Analytics with PySpark

Available actions on key/value pairs

In this section, we will be covering the following topics:

• Available transformations on key/value pairs
• Using countByKey()
• Understanding the other methods

So, this is our well-known test in which we will be using transformations on key/value pairs.

First, we will create an array of user transactions for users A, B, A, B, and C for some amount, as per the following example:

 val keysWithValuesList = Array( UserTransaction("A", 100), UserTransaction("B", 4), UserTransaction("A", 100001), UserTransaction("B", 10), UserTransaction("C", 10) )

We then need to key our data by a specific field, as per the following example:

val keyed = data.keyBy(_.userId)

We will key it by userId, by invoking the keyBy method ...