Software Testing: Concepts and Operations

11Test Oracle Design

In this chapter, we discuss the design of the test oracle, that is, the agent whose task is to observe executions of the candidate program on sample test data and to rule on whether the program does or does not behave according to its specification. We review, in turn, the tradeoffs that arise in oracle design, how to design an oracle from a relational specification, and how to design an oracle for a state-based software product.

11.1 DILEMMAS OF ORACLE DESIGN

All the test data in the world do not help us unless we have an oracle, a correct oracle, to reliably check whether the candidate program behaves according to its specification. The choice of an oracle is both critical and difficult.

• It is critical to have a reliable oracle because otherwise we run the risk of overlooking faults (if failures are not detected) and the risk of acting on faulty diagnoses (if correct behavior is reported to be incorrect).
• It is difficult and error prone to monitor the behavior of a program by having a human operator watch its inputs and outputs; but developing automated oracles poses challenges of its own, which we discuss in this section.

The general framework in which an oracle is invoked can be written as follows (where g is the program under test and S is the space of the program):

main ()
      {s = testdata;  s_init=s;
      g;   //  modifies s, keeps s_init intact
      assert(oracle(s_init,s));}

The design of the oracle is subject to the following criteria:

• Simplicity ...