This research was partially supported by NSF Grant EIA-0113539 ITR/SY+PE: “Improving the Education of Software Testers.” Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).
Abstract
Designing good test cases is a complex art. The complexity comes from three sources:
– Test cases help us discover information. Different types of tests are more effective for different classes of information.
– Test cases can be “good” in a variety of ways. No test case will be good in all of them.
– People tend to create test cases according to certain testing styles, such as domain testing or risk-based testing. Good domain tests are different from good risk-based tests.
What’s a Test Case?
Let’s start with the basics. What’s a test case?
IEEE Standard 610 (1990) defines test case as follows:
“(1) A set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.
“(2) (IEEE Std 829-1983) Documentation specifying inputs, predicted results, and a set of execution conditions for a test item.”
According to Ron Patton (2001, p. 65),
“Test cases are the specific inputs that you’ll try and the procedures that you’ll follow when you test the software.”
Boris Beizer (1995, p. 3) defines a test as
“A sequence of one or more subtests executed as a sequence because the outcome and/or final state of one subtest is the input and/or initial state of the next. The word ‘test’ is used to include subtests, tests proper, and test suites.
Bob Binder (1999, p. 47) defines test case:
“A test case specifies
the pretest state of the IUT and its environment, the test inputs or conditions, and the expected result. The expected result specifies what the IUT should produce from the test inputs. This specification includes messages generated by the IUT, exceptions, returned values, and resultant state of the IUT and its environment. Test cases may also specify initial and resulting conditions for other objects that constitute the IUT and its environment.”
In practice, many things are referred to as test cases even though they are far from being fully documented.
Brian Marick uses a related term to describe the lightly documented test case, the test idea:
“A test idea is a brief statement of something that should be tested. For example, if you’re testing a square root function, one idea for a test would be ‘test a number less than zero’. The idea is to check if the code handles an error case.”
In my view, a test case is a question that you ask of the program. The point of running the test is to gain information, for example whether the program will pass or fail the test. It may or may not be specified in great procedural detail, as long as it is clear what is the idea of the test and how to apply that idea to some specific aspect (feature, for example) of the product.
If the documentation is an essential aspect of a test case, in your vocabulary, please substitute the term “test idea” for “test case” in everything that follows.
An important implication of defining a test case as a question is that a test case must be reasonably capable of revealing information.
– Under this definition, the scope of test cases changes as the program gets more stable.
(No Ratings Yet)