High Performance Spark, 2nd Edition

Chapter 5. Testing, Validation, and Side-By-Side runs

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 11th chapter of the final book. Please note that the GitHub repo is available at https://github.com/high-performance-spark/high-performance-spark-examples.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

Errors that are caught late often require expensive re-runs of jobs, making catching errors early key to high-performance data analytics. Testing, Validation, and Side-By-Side comparisons are three key techniques for stopping errors before they hit production. These errors can lead to real-world impacts, like production outages, which more than half of people have experienced (see Figure 5-1¹), with almost 20% experiencing serious outages from their Spark pipelines. We don’t recommend hope as a strategy when it comes to software quality; for one thing, the odds are not great (worse than half).² Instead, in this chapter, you will learn how to identify and stop many kinds of errors before they impact your users.³

Figure ...