Testing Clichés Part IV – We can’t find all (important) bugs Rikard Edgren
It’s a truth that we can’t find all bugs, but is it really a truth that we can’t find all important bugs?
And it’s a cliche when used as answer to the (sincere) “why didn’t you find that bug?” question.
Testers are paid to find important information about what they are testing, and included are the big bugs, so you can fix them before the software is used in production.
Besides some good things about forthcoming test strategies, I hope you also can say: “if you have read the test plan and status reports it shouldn’t be a surprise bugs like this got away.” Well-informed risk taking is part of our business.
However, if there is a test that seems important, and would take a couple of minutes to perform, this argument is not valid. Testers (in a serious project) should try to perform all important tests that can be performed fast.
There are testing lessons to be learned if the answer is:
* we didn’t think about that scenario
* it wasn’t included in the requirements
* someone told us not to test that
* we had no idea that the customers where running that kind of configuration
* we found it, but didn’t report it well enough
* we didn’t put the bug there, and they hid it awfully well
* this type of issue can’t be found with automated tests
And there is the occasional invalid question, e.g. when the bug was reported in a good way, or when there was no testing time alotted for a late change.
Bottom line is: take the question seriously, and try to improve your testing to avoid this and other important issues (next time it will probably be another kind of issue…)
Sometimes Developers are just better at creating bugs than we are at finding them.
So additional “answers” might be:
* The code was so bad, we couldn’t expose all the bugs
* Bugs were introduced so late in the project, insufficient testing time remained
Thanks for the additions.
I guess there are mostly development lessons for “the code was so bad” example, and a testing lesson might be “get out of there”?
“…is it really a truth that we can’t find all important bugs?” What is an important bug? How important bugs are we talking about…? 🙂
I would say that using my internal rating of bugs in the context I work we can and we should find the most important bugs. We have the potential to find all the most important ones (the type I define as most important), but we cant be sure we found them all.
I completely agree that there are very valuable testing lessons to learn from faults that “slipped through”. And there are “developement lessons”. But there are also business lessons since it is common that products today are released despite known issues, and sometimes these issues are “important”.
I think there are two typical categories of “important bugs that we should find before release”
* bugs that probably would be patched if they were experienced by customers
* bugs you are embarassed to see they slipped through
I suspect it is common that projects have the potential, but not the time, to find (almost) all “important” bugs.
And yes, it is OK to release products with important defects; but wouldn’t it be fair to tell about the defects you know about?