Generating a Test Oracle from Program Documentation--work in progress

Dennis K. Peters
David L. Parnas

Software Engineering Research Group
CRL, McMaster University,
Hamilton, Ontario, Canada L8S 4K1

Abstract

A fundamental assumption of software testing is that there is some mechanism, an oracle, that will determine whether or not the results of a test execution are correct. In practice this is often done by comparing the output, either automatically or manually, to some pre-calculated, presumably correct, output [3]. However, if the program is formally documented it is possible to use the specification to determine the success or failure of a test execution, as in [1], for example. This paper discusses ongoing work to produce a tool that will generate a test oracle from formal program documentation.

In [2],[3] and [4] Parnas et al. advocate the use of a relational model for documenting the intended behaviour of programs. In this method, tabular expressions are used to improve readability so that formal documentation can replace conventional documentation. Relations are described by giving their characteristic predicate in terms of the concrete program variables. This documentation method has the advantage that the characteristic predicate can be used as the test oracle -- it simply must be evaluated for each test execution (input & output) to assign pass or fail. In contrast to [1], this paper discusses the testing of individual programs, not objects as used in [1]. Consequently, the method works with program documentation, written in terms of the concrete variables, and no representation function need be supplied. Documentation in this form, and the corresponding oracle, are illustrated by an example.

Finally, some of the difficulties encountered in the development of the test oracle generator tool are discussed.

References

  1. Antoy, S. & Hamlet, R., "Objects that Check Themselves against Formal Specifications", TR 91-1, Dept. of Computer Science, Portland State University School of Engineering and Applied Sciences, Portland, OR. 18 pgs.
  2. Parnas, D. L., "A Generalized Control Structure and Its Formal Definition", Communications of the ACM, Vol. 26, No. 8 (August 1983), pp. 572-581.
  3. Parnas, D.L. & Madey, J., "Functional Documentation for Computer Systems Engineering (Version 2)", CRL Report No. 237, Telecommunications Research Institute of Ontario (TRIO), September 1991, 14 pgs.
  4. Parnas, D. L., Madey, J. & Iglewski, M., "Formal Documentation of Well-Structured Programs", CRL Report No. 259, Telecommunications Research Institute of Ontario (TRIO), September 1992, 37 pgs.
  5. Weyuker, E. J., "On Testing Non-testable Programs", The Computer Journal, Vol. 25, No. 4 (1982), pp. 465-470.

back to Dennis Peters' homepage

Last modified: Sun 1999.01.03 at 15:33 NST by Dennis Peters