Hands-On Big Data Analytics with PySpark

Separating logic from Spark engine-unit testing

Let's start by separating logic from the Spark engine.

In this section, we will cover the following topics:

• Creating a component with logic
• Unit testing of that component
• Using the case class from the model class for our domain logic

Let's look at the logic first and then the simple test.

So, we have a BonusVerifier object that has only one method, quaifyForBonus, that takes our userTransaction model class. According to our login in the following code, we load user transactions and filter all users that are qualified for a bonus. First, we need to test it to create an RDD and filter it. We need to create a SparkSession and also create data for mocking an RDD or DataFrame, and then test the ...