Evidence for an Information Mismatch

We compared the results from the developer and reporter surveys to find out whether they agree on what is important in bug reports.

Results from the survey among reporters (215 consistent responses by Apache, Eclipse, and Mozilla reporters)

Figure 24-2. Results from the survey among reporters (215 consistent responses by Apache, Eclipse, and Mozilla reporters)

First, we compared the information developers use to resolve bugs (question D1) and the information reporters provide (R1). In Figure 24-3, items in the left column are sorted decreasingly by the percentage of developers who have used them, and items in the right column are sorted decreasingly by the percentage of reporters who have provided them. Lines connect the same items across columns and indicate the agreement (or disagreement) between developers and reporters on that particular item. Figure 24-3 shows that the results match only for the top three items and the last one. In between there are many disagreements, and the most notable ones are for stack traces, test cases, code examples, product, and operating system. Overall, the Spearman correlation[33] between what developers use and what reporters provide was 0.321, far from ideal.

Used by developers versus provided by reporters

Figure 24-3. Used by developers versus provided by reporters

Next, we checked whether reporters provide the information that is most important for developers. In

Get Making Software now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.