Making Software

Phase 1: Overall Survey

There were three specific purposes in the original overall survey:

• To determine, generally, what kinds of problems were found (which we report here), as well as, specifically, what kinds of application-specific problems arose during the preparation of this release (which we do not report, because of their lack of generality)

• To determine how the problem was found (that is, in which testing phase)

• To determine when the problem was found

One of the problems encountered in any empirical survey study is ensuring that the survey is in the “language” of those being surveyed. By this “language” we mean both the company- or project-specific jargon that is used but also the process that is being used. You want the developer surveyed to clearly understand what is being asked in terms of his own context. Failing to understand this results in questions about the validity of the survey results. To this end, we used developers to help us design the survey, and we used the project jargon and process to provide a familiar context.

Summary of Questionnaire

The first phase of the survey questionnaire had two main components: the determination of the category of the fault reported in the MR and the testing phase in which the fault was found. In determining the fault, two aspects were of importance: first, the development phase in which the fault was introduced, and second, the particular type of the fault. Since the particular type of fault reported at this stage of the survey tended ...