Jerry Weinberg has, in his book ”Perfect Software and other illusions about testing”, expressed a very important observation, namely who is responsible for pinpointing the bug. The tester finds the bug, tries to reproduce it, then adds as much information that he/she has such as log files, configurations, test data and so on. When you estimate time for testing I think you most often consider the time that you do to find the bug and then make a report. As Jerry points out, the time it takes to pinpoint the exact location is not taken into consideration. It might also be unclear who has this responsibility.

This might not be an issue where there is high testability, when it takes little time to report the bug and when the actual pinpointing takes little time. When testing complex systems (a system consisting of several sub system, different hardware, many interfaces, a vast amount of ways to get information on what actually went wrong), the time for pinpointing can be a factor that is underestimated and that can really be a major issue in a project. Reproducing a bug can take several hours or possibly days. If you, after having reported the bug, are asked by the developer to add extra or missing information to the bug it might cost you days to get into the same erroneous state. In some cases it might be close to impossible because of missing equipment or missing/broken units. When it takes this long time it is inevitable that a conflict grows in the project. It is also inevitable that the testers start to doubt how much time they should spend on bug reporting (in the worst case scenario consider not doing it) and if they can help developers.

How do you handle pinpointing? Is there any need to increase testability to make it easier to report bugs and get information from the sysem? How have you discussed and communicated this within our projects? Is your testlead and project manager aware of this time sink and uncertainty?

If you want to taste some of Jerry Weinbergs knowledge I recommend reading this book.

I use Jerry Weinberg’s definition of quality: “Quality is value to some person”; and I use Cem Kaner’s extension to the definition so that it becomes “Quality is value to some person (that matters)”…

I.e. quality is inherently subjective. And there are a lot of persons that are affected by software that we produce… With this in mind it becomes hard for a tester to stay focused when there are so many persons with opinions that could matter; but if we can find out “who matters” we decrease the number of possible values to care about. Still, this will leave us with several important values that need to be taken into account when testing the product.

So how can we testers deal with that?

You could do a role play when testing and put on someone’s hat during the test session; or you could let real users test the product and let them have a say about what they find.
But for a skilled tester it is more about being multidimensionally subjective and think as several persons at the same time.

This means that a lot of values, beliefs and preferences are taken into account which might matter. Not as an average, but as several independent quality dimensions that has (more or less) importance. The hard thing is to know when a value is threatened and for which (type of) person that is affected; and if this matters at all.
I.e., it is a matter of questioning “is there a problem here?” constantly and try to pair a potentially threatened value with its corresponding person. And if this problem threatens a value for some person that matters, we have found a bug. This corresponds to the definition of bug from Cem Kaner “A bug is something that threatens the value of a product”.

Much of this happens automatically for many of you skilled testers out there; when I thought of it recently I realized that this is something I do more and more and hopefully I am improving this skill each day. This is a great skill to have when testing software!

Anyone having any thoughts on this?
Have you experienced this yourself?
If not, does it sound like an interesting thing to examine? Would this be helpful to you?

Cheers,
Henrik

Update 2009-09-14: According to comment from Michael Bolton, see below, the quotes that I said belonged to Cem Kaner are both quotes from James Bach. I apologize for referencing wrong person.