The Art of Application Performance Testing, 2nd Edition

Chapter 5. Interpreting Results: Effective Root-Cause Analysis

Statistics are like lamp posts: they are good to lean on, but they often don’t shed much light.

Anonymous

OK, so I’m running a performance test—what’s it telling me? The correct interpretation of results is obviously vitally important. Since we’re assuming you’ve (hopefully) set proper performance targets as part of your testing requirements, you should be able to spot problems quickly during the test or as part of the analysis process at test completion.

If your application concurrency target was 250 users, crashing and burning at 50 represents an obvious failure. What’s important is having all the necessary information at hand to detect when things go wrong and diagnose what happened when they do. Performance test execution is often a series of false starts, especially if the application you’re testing has significant design or configuration problems.

I’ll begin this chapter by talking a little about the types of information you should expect to see from an automated performance test tool. Then we’ll look at real-world examples, some of which are based on the case studies in Chapter 4.

For the purposes of this chapter you can define root-cause analysis as a method of problem solving that tries to identify the root causes of (performance) problems. This is distinct from a causal factor that may affect an event’s outcome but is not a root cause. (Courtesy of Wikipedia.)

The Analysis Process

Depending on the capabilities ...