Scala and Spark for Big Data Analytics

Testing Spark applications

There are many ways to try to test your Spark code, depending on whether it's Java (you can do basic JUnit tests to test non-Spark pieces) or ScalaTest for your Scala code. You can also do full integration tests by running Spark locally or on a small test cluster. Another awesome choice from Holden Karau is using Spark-testing base. You probably know that there is no native library for unit testing in Spark as of yet. Nevertheless, we can have the following two alternatives to use two libraries:

• ScalaTest
• Spark-testing base

However, before starting to test your Spark applications written in Scala, some background knowledge about unit testing and testing Scala methods is a mandate.