Beautiful Testing

Test Case Effectiveness

What is so beautiful about testing?

We confess that sometimes it is difficult to see the beauty in software testing. It may be that software QA suffers from a lack of respect and funding because it lacks the superficial glamour enjoyed by other aspects of software development.

While cruising Istanbul’s Bosporus channel with other members of the GNOME community last summer, we were asked whether we used Dogtail or Linux Desktop Testing Project (LDTP) for our automated testing. We knew we had evaluated these tools, but we were embarrassed to admit that we didn’t know exactly why we didn’t rely on them. The exact reason had to do with stability, performance, and the probe effect. But in general, automated test tools are never sufficient for desktop or GUI testing. Why is this? Because it is impossible to simulate every possible combination of user actions on a graphical user interface. Think of how many possible ways you could interact with 10 buttons on a screen. There are more than three million possible different orders in which you could click the buttons, and you could single- or double-click on each button. You could even drag after clicking. Fortunately, some of the low-level behaviors of buttons on a GUI are in well-tested libraries, and other behaviors are prevented. But you can see where a brute-force automation testing methodology is apt to fail.

Fortunately, real users can provide a representative sample of likely workflows that can be fed into test ...