Taking Testing Seriously

CHAPTER 9Test Reporting (Without Pretentious Metrics)

—By Paul Holland

You have likely seen test reports containing charts, graphs, and metrics that claim to show the status of testing. Some claim to show the status of the product. I think the vast majority of those metrics are not only useless in showing the status of a project but also dangerous to use. The danger comes from the illusion that they are providing actionable information. For decades, stakeholders have been making decisions on whether or not to release a product based, at least partially, on these dangerous metrics. In this chapter, I propose alternatives to using these metrics that show actionable, useful information that stakeholders can use to make informed decisions.

Metrics Are Nothing Without a Story

Test management tools generate a lot of metrics based on test cases. This is a big problem for the industry, however, because test cases are not a reliable unit of measure. It's like counting backpacks or plastic bags to determine the status of a business.

Worse, many testers don't make any coherent report to go with the metrics. Their companies simply assume that if all the automated checks are green, testing is done, and the product is okay to deploy.

To demonstrate the issue, let's look at two hypothetical projects that are similar in size, complexity, and importance to the company (see Figure 9-1).