Making Software

The First Experiment: Testing Pattern Documentation

The first question we asked was whether merely documenting design patterns would improve programmer performance. The idea was to give two sets of subjects the same program to modify, but one group would get slightly extended program documentation pointing out the patterns present; the other group would get normal documentation, but no pattern information. This experiment would thus test only the presence or absence of pattern documentation, not the effect of patterns. We chose this approach partly because when designing the experiment in 1996, we hadn’t figured out how to construct two versions of a program that were equivalent, but with only one containing patterns. In a later experiment we were able to overcome this hurdle. But testing pattern documentation alone still proved useful because it gave us a first indication of whether patterns could be effective. If documenting patterns alone would increase performance, even bigger effects could be expected when comparing programs with patterns against those without.

Design of the Experiment

The experiment question was this: does it help the software maintainer if the design patterns in the program code are documented explicitly (using source code comments), compared to a program without explicit reference to design patterns? The experiment question was refined into the following two hypotheses:

Hypothesis 1

Documentation of design patterns speeds up pattern relevant maintenance tasks ...