Lessons Learned in Software Testing: A Context-Driven Approach

Chapter 5. Automating Testing

Robots that make you breakfast. Flying cars for commuters. That's science fiction. But software can do anything. So, why not make software test software? The reasoning goes, if one computer can do the work of three million mathematicians using sticks and sand, then surely one computer is worth an army of human testers. Indeed, test automation is an exciting idea that holds great promise. But beware. Automating some of your testing might or might not be helpful. Automation can save time, speed development, extend your reach, and make your testing more effective. Or it can distract you and waste resources.

Your investment in test automation is valuable to the extent that it helps you achieve your mission. The role of testing is to gain information. What information is your automation providing?

Automation efforts have been spectacularly successful for some groups but have left others unhappy and frustrated. Some of the failing groups have deluded themselves and their management into thinking that the work they put into automation yielded something that was helpful.

Use automation when it advances the mission of testing. Evaluate your success at automation in terms of the extent to which it has helped you achieve your mission.

Reviewers of early drafts of this chapter have separately urged us to emphasize some contradictory points:

Design your tests first, before deciding which to automate.

This prevents you from falling into the trap of automating tests that ...