Testing Clichés Part I – Expected Results Rikard Edgren

Sometimes it is said that each test case must have an expected result, or even worse, that each step of a test case must have an expected result.
This is the extreme of scripted testing that I dislike for two reasons:
* It takes a lot of time to write and follow detailed test cases; time that is better spent testing things you haven’t had the time or enough information to thoroughly document.
* It takes away the freedom of the tester to try things a bit different, so we will learn less, narrow our thinking, have less chance of finding new, interesting information, and it will be boring to test.

I understand that the reason for the details is that you want to be sure exactly what has been tested.
But the problem with this reasoning is that in software testing, there are so many things that aren’t tested. You are not testing on the exact same data that your customers use, you are not testing on the exact same hardware and interacting software, and you are not testing in the exact same way as the customers will use your software.
So we can be sure which tests have been run, but that doesn’t necesarily mean it will work for the customers.

This is an example where (Bad) Process is way too much more important than Content.

I prefer a lot of one-liner test ideas, that either can be used in vague test cases or as starting point for Exploratory testing.

3 Comments
Rob Lambert March 23rd, 2009

Hi,

Good post. I like it a lot. I feel the same way about test cases and expected results. I once worked somewhere where the test cases were limited to 10 steps only and wouldn’t save if there were no expected results! Process overload. I actually did a blog posting in a similar vain to this one. http://tinyurl.com/dbkcfj

Cheers
Rob..

Henrik Emilsson March 23rd, 2009

There are so many unexpected results to expect from a good test that you cannot expect anything but the unexpected.

Martin Jansson March 23rd, 2009

I’ve been forced in having Expected Result to be included in a test case as well. It is extremely bad. In those cases I’ve just added a few lines of tips what you can look for. Just adding a bit of your experience as a tester what you think could happen.

It might be dangerous doing even that, because you will narrow the testing eye so that you will be influensed in not looking at other places. Still, if you know that there is an infinite number of places to look for bugs and that what you see in the “expected result field” as tips, it might not be that bad after all?

Would this be measurable?