Review of properties in Kaner’s What is a Good Test Case? Rikard Edgren

One of Cem Kaner’s many classic writings is “What is a Good Test Case?
It is a very good article, well-spent time for anyone involved in software testing.
But when writing about test ideas, I started to realize that the list of properties for good test cases isn’t perfect, for me.
So it’s time for some criticism of this part of the professor’s article.

His list of attributes for good test cases (for the context “Tests Intended to Expose Defects”) goes something like this:

  • Powerful – a test is more powerful if it is more likely to find bugs
  • Yield significant results – the issues found are important (to stakeholders)
  • Credible – the actions in the test are realistic (not corner cases “noone woud do”)
  • Likely – the test reflects how customers actually will be using the product
  • Easier to evaluate – ease of telling if the test passed or failed
  • Useful for troubleshooting – so it is easy to find out what went wrong during the test
  • Informative – regardless of pass/fail status, you get valuable information from the test
  • Appropriately complex – if there are many bugs in the software, the test might fail too quickly
  • Giving insightful information – the test might not render bugs, but other important information

Test cases can be seen as a broad spectra, from “classic” test cases with exact steps and expected results, to vague, one-liner test ideas (that also could be ongoing, unorthodox or unverifiable.)
So I have to disagree with “Easier to evaluate”, it’s a valuable property in many situations, but not in so many that it should make it on this list. But it’s back on the list if it is named something like “Hints for Evaluation”.

But now to the most interesting part of reviewing: what’s missing on the list?
There are many more things that could be important, a quick search gave:

These are good things, but not needed on my list (and I’d prefer Fast to execute as name instead of Economical.)
But I have two other things I would like to add

  • Easy to understand – to enable reviewing by different types of people (more likely to happen for one-liners)
  • Enables serendipity – the test is rich in the sense it has possibilities of finding issues other than the ones the test case is aiming for

The serendipity can either be covered inside the test case, or by allowing variations, or by putting freedom and responsibility on all testers to deviate and look at more things, if deemed worthwile, while executing the test cases.

…and always remember Kaner’s advice that “Test cases can be “good” in a variety of ways. No test case will be good in all of them.

5 Comments
Martin Jansson May 19th, 2010

I think Economical is more in line with Yields high value. Fast to Execute is a separate, but good thing. Easy to understand goes in line with Easy to communicate, which is essential if you want people to be able to review it, as you say.

A one-liner is most often easier to communicate and understand than a big test case with lots of repro-steps and other content. I promote to do the review on the test ideas then let them grow into something else if they are good enough.

Rikard Edgren May 19th, 2010

Agreed!

Jon Tilt September 13th, 2010

Thanks for the comment on TestingBlues.com, I like the look of your blog, looks like you guys are our Scandinavian equivalent :)

Added thentesteye.com to our blogroll in recognition.

Cheers
Jon

Rikard Edgren September 13th, 2010

I hope thetestingblues.com also have felt the positive synergic effects when several people blog on the same subjects and comment on each others posts, and hopefully take them several steps further.
I’m surprised there aren’t more collaborations like this.

Rikard Edgren September 22nd, 2010

Cem Kaner has more on this: a list with 18 categories in slide 154 of
http://www.testingeducation.org/articles/ExploratorySkillsQAI2007.pdf