thoughts from the test eye

One of Cem Kaner’s many classic writings is “What is a Good Test Case?”
It is a very good article, well-spent time for anyone involved in software testing.
But when writing about test ideas, I started to realize that the list of properties for good test cases isn’t perfect, for me.
So it’s time for some criticism of this part of the professor’s article.

His list of attributes for good test cases (for the context “Tests Intended to Expose Defects”) goes something like this:

• Powerful – a test is more powerful if it is more likely to find bugs
• Yield significant results – the issues found are important (to stakeholders)
• Credible – the actions in the test are realistic (not corner cases “noone woud do”)
• Likely – the test reflects how customers actually will be using the product
• Easier to evaluate – ease of telling if the test passed or failed
• Useful for troubleshooting – so it is easy to find out what went wrong during the test
• Informative – regardless of pass/fail status, you get valuable information from the test
• Appropriately complex – if there are many bugs in the software, the test might fail too quickly
• Giving insightful information – the test might not render bugs, but other important information

Test cases can be seen as a broad spectra, from “classic” test cases with exact steps and expected results, to vague, one-liner test ideas (that also could be ongoing, unorthodox or unverifiable.)
So I have to disagree with “Easier to evaluate”, it’s a valuable property in many situations, but not in so many that it should make it on this list. But it’s back on the list if it is named something like “Hints for Evaluation”.

But now to the most interesting part of reviewing: what’s missing on the list?
There are many more things that could be important, a quick search gave:

• Ease of maintenance, Ease of creating variations of it (Shrini Kulkarni)
• Accurate, Economical, Reusable, Tracable, Self-cleaning (Dianne L. Runnels)

These are good things, but not needed on my list (and I’d prefer Fast to execute as name instead of Economical.)
But I have two other things I would like to add

• Easy to understand – to enable reviewing by different types of people (more likely to happen for one-liners)
• Enables serendipity – the test is rich in the sense it has possibilities of finding issues other than the ones the test case is aiming for

The serendipity can either be covered inside the test case, or by allowing variations, or by putting freedom and responsibility on all testers to deviate and look at more things, if deemed worthwile, while executing the test cases.

…and always remember Kaner’s advice that “Test cases can be “good” in a variety of ways. No test case will be good in all of them.”