Beautiful Testing

Reporting

From management’s point of view, test results may be the most important artifact of the test team. Current and relevant test results are another key aspect of an automation system, and it is critical that test results are current, accurate, and easily available.

The role of an automation system in reporting is to ensure that the items in the previous paragraph are possible. Tracking and tagging of test results is imperative. For any new failures (automatic failure analysis should filter errors found in previous testing), information about the failure such as call stacks and logfiles need to be readily and quickly available so that diagnosis of the issue by the tester is as efficient as possible.

The Truth About Test Results

Every team I know of that runs automated tests also tracks test results closely. Most of these teams have goals to reach or exceed, and a specified pass rate by the end of a milestone or by the time of release. For example, they may have a goal of passing 95% of their tests, or passing 100% of their “priority 1” test cases.

When teams tell me this, I always follow up by asking them what they do if they don’t reach their goal. I ask, “What do you do if you’re at a milestone exit, and your pass rate is only 94%?” They inevitably answer, “Oh, it depends on the failure. If the errors blocking us from reaching our goal don’t meet the bar, we go ahead and move on.” This, I suppose, is better than telling me that they just stop running that test (which is, unfortunately, ...