High Performance Spark, 2nd Edition

Appendix C. When Not to Use Spark

Apache Spark is an amazing tool for handling large-scale data with massive parallelism, but not all tasks are well suited to data parallelism. In this appendix you’ll learn how to quickly look at a problem and see some possible warning signs that Spark might do more harm than good as well as some other tools you might want to consider in these cases. While some situations are more obvious, like coalesce(1), collect, and toLocalIterator, not all cases where data parallelism doesn’t help are quite so direct. The most conclusive way to tell if using Spark is beneficial is to write your program with and without Spark and compare the performance, but that’s normally impractical, so instead let’s explore some simple heuristics to sort it through.

Tip

If you have an existing Spark job and you find that it’s frequently executing a single partition or is spending a substantial amount of time without any tasks running on the cluster, using Spark might not be worth it. If you don’t want to refactor your code to rip out Spark, you can try switching it from distributed mode into local.

We tend to think of the biggest downside of using Apache Spark over other tools is the distributed overhead. While we can reduce some of that downside by using local mode, and Spark has continued to reduce its overhead, there are other downsides that come from using Spark. Most, but not all, local tools have better debugging experiences than distributed-capable tools like Apache ...