Beautiful Testing

Case Study of the OpenSolaris Desktop Team

The desktop QA team is responsible for the quality of all the desktop applications included with OpenSolaris. These applications include open source projects from many communities, including GNOME, Mozilla, Compiz, Pidgin, etc. OpenSolaris is on a six-month release schedule, but a development build is released to the community every two weeks. Due to the frequency of releases and to ensure high quality, the desktop QA team adopted a formal test process with a focus on functional testing. The QA team was responsible for the test plan and the design of test cases. Many of the test cases come from open source communities. For example, most of the test cases for Firefox are from the Mozilla community, which uses a test case management tool called Litmus.

For most of the desktop applications, although 100% of the test cases are executed, there are still bugs filed out of the normal QA test cycle. The QA team tracks the TCE trend at every build in order to measure and improve the effectiveness of the test cases over the course of development of an OpenSolaris release. The following sections outline an example of how to measure and improve the test cases’ effectiveness.

Assumptions

For the OpenSolaris desktop project, we classify all bugs submitted by non-QA engineers as test escapes, excluding the following:

• Enhancement

• Localization (L10N) OS-related

• Hardware

• Device drivers

• Globalization (G11N)

• Build

• “Cannot reproduce”

• “Not a bug”

• Duplicates

The ...