Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing

3.5. "There's a Lesson to Be Learned Here...": Test Case Incremental Improvement

No matter how good a job I do on my first crack at developing test cases and suites, my test systems always have holes. Budget and time pressures, along with human mistakes, result in incomplete testing. Some test cases won't cover conditions they should, or some test suites won't include important test cases, or the overall test system won't have all the test suites it needs.

In the case of most consumer software and hardware products, a few omissions won't prove fatal to the product, provided you are careful to test the critical features. (The issue of test escapes takes on a different meaning when you are dealing with safety-critical systems such as traffic-light, medical, or power-plant control computers or with situations that involve people's privacy and security. A higher standard of quality is required in safety- or mission-critical systems.) Nonetheless, I attempt to improve my test system for each subsequent release. A few techniques for doing this involve responding to the failures in the shipping product, incorporating the practices of others into your test system, and using exploratory testing.

3.5.1. Responding to Failures

One obvious way of enhancing a test system is to plug gaps that come to light when some poor customer or user is whacked in the forehead by a test escape. Of course, this doesn't mean that you or anyone else in your company will celebrate test escapes; you'd rather ...