Making Software

Lessons Learned

This series of experiments was a tremendous learning experience. The experiments were difficult to construct. Adding pattern documentation was easy enough once we thought of it, but constructing “fair” comparisons for pattern programs was hard, because the alternative solutions always looked very much like the pattern solutions. The experimenters were already locked into thinking in patterns! We made progress only after we accepted that the alternative solutions could be simpler and less flexible and that that was exactly what should be tested. Finding ways to capture communication stymied us for a long time. After a lot of reading about experimental design, we decided to use protocol analysis. But what to do with the protocols? The concept of the ideal communication line and comparing it with actual ones was a stroke of genius by Barbara Unger (see the Acknowledgments section). Preparing, running, and analyzing the experiments was unexpectedly time-consuming (three PhD dissertations’ worth), but we did it at the right time, when the topic was fresh and subjects easy to find. In the end, we were thrilled with the results.

The book by Christensen [Christensen 2007] about experimental design was a godsend. It is in its 10th edition and covers all the major topics of experimental methodology.

I recommend to always use a counterbalanced design if subjects do more than one task. Counterbalancing does not cost anything except a bit of organization, and the ability to check ...