High Performance Spark, 2nd Edition

Chapter 11. Testing, Validation, and Side-By-Side Runs

Errors that are caught late often require expensive reruns of jobs, so catching them early is key to high performance data analytics. Testing, validation, and side-by-side comparisons are three key techniques for stopping errors before they hit production. These errors can lead to real-world impacts, like production outages, which more than half of people have experienced (see Figure 11-1), with almost 20% experiencing serious outages from their Spark pipelines. We don’t recommend hope as a strategy when it comes to software quality; for one thing, the odds are not great (worse than half).¹ Instead, in this chapter, you will learn how to identify and stop many kinds of errors before they impact your users.²

We discuss unit and integration testing, validation, and side-by-side runs separately, even though they could all be (broadly) considered a form of testing. In this book, we consider testing mainly the offline process (automated or manual) of verifying that our software behaves as desired. Validation (sometimes called audits) is the online process of ensuring that our job is working as expected. Hidden side-by-side runs are a special kind of testing or validation that can be used when making changes to a job that you don’t expect to impact its output, including performance tuning.

Figure 11-1. Spark production outage question ...