Who does the pinpointing? Martin Jansson
Jerry Weinberg has, in his book “Perfect Software and other illusions about testing”, expressed a very important observation, namely who is responsible for pinpointing the bug. The tester finds the bug, tries to reproduce it, then adds as much information that he/she has such as log files, configurations, test data and so on. When you estimate time for testing I think you most often consider the time that you do to find the bug and then make a report. As Jerry points out, the time it takes to pinpoint the exact location is not taken into consideration. It might also be unclear who has this responsibility.
This might not be an issue where there is high testability, when it takes little time to report the bug and when the actual pinpointing takes little time. When testing complex systems (a system consisting of several sub system, different hardware, many interfaces, a vast amount of ways to get information on what actually went wrong), the time for pinpointing can be a factor that is underestimated and that can really be a major issue in a project. Reproducing a bug can take several hours or possibly days. If you, after having reported the bug, are asked by the developer to add extra or missing information to the bug it might cost you days to get into the same erroneous state. In some cases it might be close to impossible because of missing equipment or missing/broken units. When it takes this long time it is inevitable that a conflict grows in the project. It is also inevitable that the testers start to doubt how much time they should spend on bug reporting (in the worst case scenario consider not doing it) and if they can help developers.
How do you handle pinpointing? Is there any need to increase testability to make it easier to report bugs and get information from the sysem? How have you discussed and communicated this within our projects? Is your testlead and project manager aware of this time sink and uncertainty?
If you want to taste some of Jerry Weinbergs knowledge I recommend reading this book.
This is a very interesting area, where I think it’s almost impossible to be at the exact right level. It’s not only the time issue, it is also a question of learning, that happening when pinpointing, and can be very useful knowledge, both for testers and developers.
One answer to the “Who” question could be “the group that has most available time” or rather “the group that is the least over-booked”.
When forming team size, it could be argued “with one more tester we have resources to do more of the pinpointing.”
When deciding scope, it can be good to put more effort on logging, in order to save time when those tricky bugs appear.
The pinpointing issue can also be fuel for conflicts, e.g. accusations that testers don’t put enough effort on the right information in the bug reports.
If you have a situation where testers don’t like to work with certain features, due to too much extra-investigations for every bug, then you have a problem where resolving the pinpointing ownerships can be part of the solution.
I recently read Michael Bolton’s blog about why testing takes so long http://www.developsense.com/blog/archive/2009_11_01_archive.html
I am in a situation now where we find and analyse too many simple bugs so that more advanced testing is severly limited. So my problem is even bigger than just pinpointing – it is also about not enough developer testing.
At work – we need to take this discussion in january together with pinpointing issues. We do a lot today because we can and bugs get fixed faster but at the same time we undermine our possibilities of doing a good job. I do not mind helping out as much as I can but it has a price – that is what our discussion is going to be about.
I really like Michael’s article as well. I agree fully that finding cheap bugs makes this even more critical.
I’ve not seen this as an issue in smaller organsations, it is usually handled one way or the other without getting into the spotlight.
I think this is more common in larger organisations where information exchange is harder and more complex. If there are several layers to the test organisation it becomes even more complex.
Either way, I think this is very underestimated in project planning and test planning.
Good questions and thoughts Martin!
I guess that this depends as you say on the type of product and project that you work with.
In my experience, having a very complex environment you don’t want to find bugs that will give you two more days of “investigating” or pinpointing. Unless you get paid by the hour of course! 🙂 And a part of this is that the project is too large for any team spirit to grow, so you end up doing something boring and blame someone else for not doing their job; the same way that “someone else” feels when they are assigned to do the pinpointing.
At one time I got in the middle of a trench warfare that lasted for several weeks without anything creative happened at all… I think that the bug was closed as not repro after 6 weeks and no one really cared.
Working in semi-complex environments I have come a long way by discussing with the developers on how much information they would like to get. And in many cases they are just fine with a link to the log files, a one-liner error message and the timestamp where the error occurred. Experienced developers knows where to look anyway. Others need more assistance.
But as always, isn’t this a question for the project manager? Where should the project spend the money?
I would suggest discussing this with the project manager and come up with a model that would work in your project. Should we make the product more testable? Or even stable? Should we increase the test team? Would it be good to have a dedicated problem solver in the team? Or just someone that helps the project with pinpointing important problems? How about a whole task force that specializes in investigating?