Symbian for Software Leaders: Principles of Successful Smartphone Development Projects

Chapter 10. Managing testing

Beyond complete testing

For small enough projects, it's possible to make do without structured testing. An informal set of tests is sufficient to provide a good indication of whether the software meets its intended requirements.

For larger projects, such as feature phones, testing becomes much more formal. There are a large number of test cases, called "system tests", covering the full set of functionality of the phone. The phone can be exhaustively tested for defects. Many companies follow the rule that a feature phone cannot be released to the market, if even one of the system tests fails. The feature has to be fixed before the phone is released.

However, smartphone projects are yet larger again, with an open-ended functionality set. This means that a new approach to testing is needed. You can't adequately test a smartphone by just repeating the same methods used to test a feature phone. It's another case when simply "working harder" is insufficient. Any idea of "complete testing" isn't feasible for a smartphone project:

• There is an unbounded number of different combinations of applications that can be running at the same time (including add-on third-party applications that have not yet been written!)

• Each different combination of applications, in principle, throws up new potential interaction issues.

Not only is "complete testing" infeasible, trying to execute it is an inefficient (wasteful) policy:

• Many of the potential tests are essentially duplicates ...