Hands-On Big Data Analytics with PySpark

Extending an RDD

This is a simple test that has a lot of hidden complexity. Let's start by creating a list of the record, as shown in the following code block:

class InheritanceRdd extends FunSuite {  val spark: SparkContext = SparkSession    .builder().master("local[2]").getOrCreate().sparkContext  test("use extended RDD") {    //given    val rdd = spark.makeRDD(List(Record(1, "d1")))

The Record is just a case class that has an amount and description, so the amount is 1 and d1 is the description.

We then created MultipledRDD and passed rdd to it, and then set the multiplier equal to 10, as shown in the following code:

val extendedRdd = new MultipliedRDD(rdd, 10)

We are passing the parent RDD because it has data that was loaded in another RDD. In this ...