High Performance Spark, 2nd Edition

Appendix F. The Spark Web UI: Debugging and Optimizing Your Jobs

The Spark Web UI is one of the most powerful yet underutilized tools in a data engineer’s arsenal. While we’ve referenced it throughout this book for checking partition sizes, monitoring spill behavior, and examining cached RDDs, this appendix provides a comprehensive guide to understanding and effectively using every aspect of this built-in monitoring interface. Rather than presenting a dry reference manual, we’ll learn to use the Web UI by debugging a real performance problem, a scenario that every data engineer encounters sooner or later. In this section, we hope to give you the tools to think about how to interpret what the Web UI is telling you and, more importantly, how to translate those observations into actionable performance improvements.

A Real-World Performance Problem

The first time many of us develop a big data application with Apache Spark, something unexpected happens: a simple job that should complete in seconds takes minutes, or never finishes at all. The culprit is often small files. We partition data incorrectly and accidentally write millions of tiny files to Amazon S3 or HDFS. The code looks correct, and the logic is sound, but the performance is terrible.

Let’s re-create this exact scenario with a simple Spark script that reads sales data, filters by quantity, and writes the results partitioned by quantity:

val df = spark.read.load("~/data/store_sales")

val dfFiltered = df.filter(df("ss_quantity" ...