Resolved as Not Repro Henrik Emilsson
Lets say that you have a bug system; and for each bug you have the two fields “State” and “Resolution” where the following values are valid:
State: New, Assigned, Resolved, Closed.
Resolution: Fixed, Invalid, Won’t fix, Duplicate, Not Repro.
Further, you have a field where a product version number should be entered; i.e., the earliest version number of the product where the bug can be reproduced in.
Now, lets say that you report a serious crash bug that is reproduced in the latest build of the product e.g., version 1.1.2. (Oh, I forgot to mention that you work as a software tester at a software company and this happens during development of the upcoming release of your product.)
Two weeks later the bug is resolved by the assigned developer; the State and Resolution is set to “Resolved – Not Repro” and the following comment is added by the developer in the bug:
“I cannot reproduce this anymore (code is same as in build 1.1.4). This might have been fixed when other bugs in the same area were fixed. Resolving as Not Repro…”
So what do you think?
Do you think that this bug should have been resolved as Not Repro?
If not, what resolution would you have chosen?
Would it make any difference if the resolution “Works for me” had been a valid resolution instead of “Not Repro” (as in Bugzilla)?
Any other thoughts on this?
I think that a Not Repro bug can only be not repro in the same build, configuration etc. If a developer “think” it is fixed later on by other bugfixes or some such, the bug has been Fixed until proven otherwise.
Worksforme could be set in the same build, configuration etc.
Still, you can always leave a comment in the bug report that it seems to work in a specific release.
Not sure what bugzilla states in their help regarding this, but the above is what I preach.
If it was reproducible in 1.1.2 and no longer is, then either someone fixed it or it was made a non-issue by virtue of a redesign. I’d either mark it “Fixed” and find out what files may have changed to fix it, or mark it “Not Repro- Redesign”.
Many factors here – there can be several inter-connected problems that are not traced to a similar root cause (especially if they occur on incremental builds).
If this is a fast-turnaround project or the deliveries are intermediate then it’s very possible for the project manager/responsible to say that if the functionality isn’t degraded or that the fault isn’t reproducible then it’s “Not repro”.
This can be quite common in load/performance problems – their resolution can touch several areas and indeed make other issues “disappear”. In actual fact it usually mights that several issues are connected – eg having a similar “ancestor” problem (the fix to this changes the behaviour downstream meaning problems may not occur, problems may be different or new problems/issues emerge).
Five years ago I would have seen the described scenario as something completely wrong; and it would have hurt me as a bug reporter as it looks like an invalid report. I.e., it would have affected my credibility as bug reporter.
Now when I think of it, it does not have that big impact on me as before.
I kind of like the “Works For Me” resolution, because that is not as controversial as “Not Repro”; it merely say that “this seem to work for me when I test it on build xxx”. Note that you need to describe which version of code you tested on.
Ideally I would prefer that the bug was Resolved as Fixed and that the analysis done by the developer revealed that “it was fixed when bugs 123 and 234 were fixed”.
But sometimes this analysis needs to be thorough and therefore takes more time (higher cost) than could be motivated; so the “Resolved by Not Repro” or “Resolved by Works For Me” could suffice. As long as all participants knows the conditions.
I have faced this situation many a times. It really bugs me to accept the ‘
Resolved – Not Repro’ state when it was very much Reproducible. If a defect is moved to ‘Not Repro’ state, then it should be on that particular build only. If we pick up a later build and then realise, it is fixed, then the status needs to be ‘Resolved – Fixed’. we also need to investigate what fixed that issue without developers explicitly fixing it if at all they claim that they did not touch that part of the code. Regression testing becomes very important for such defects as there could be a bigger risk hidden on the inside.
I once worked with a developer who would take about 3-4 testing builds to start looking into an issue. She would ideally fix the issue and change the state to ‘Resolved – Not Repro’. One day, I picked up the defective file(for a particular defect) and the same file in the latest build that somebody has fixed it. She confessed that she has fixed that issue. Later on, I got to know that she was worried to have too many ‘Resolve – Fixed’ defects because her Manager tracks the number of issues filed by Testing Team on the module she was working on. Metrics scare the hell out of everyone I guess.
Regards,
Pari – http://curioustester.blogspot.com
Pari said:
“we also need to investigate what fixed that issue without developers explicitly fixing it if at all they claim that they did not touch that part of the code. ”
I am not sure that you /need/ to do the investigation.
In some cases, yes, it is really important that you investigate this; but in many cases it might be fine to accept that this has happened. What matters is that the bug has been fixed!
Another important thing, as you touch in your comment, is that metrics should not be a basis for measuring the efficiency of both testers and developers.
———————————————-
I have experienced developers that have a tendency to Resolve as Not Repro too often; I have experienced developers that have “fixed” bugs that really was not reproducible in the first place; and I have experienced developers that has Resolved as Fixed but they clearly have not touched any code.
These /developers/ are all better candidates for investigation than the /bugs/ themselves! 🙂
Same goes with testers that report too many Not Repro/Invalid/Duplicate bugs.
I don’t think it means that the developer didn’t appreciate the tester’s work. It just meant that the dev checked and the bug didn’t occur anymore. Sure, maybe it’s “fixed”, but then again maybe not. Maybe it still repros for the tester even on that newer build.
The same thing happens as with any resolved bug: the tester verifies that the bug no longer exists, maybe reviews code changes that might have been involved, looks for similar problems in the rest of the codebase, tries to determine whether there’s yet another bug exposed/introduced because the original one is out of the way, etc. Does it really matter what the resolution was as long as the dev communicated clearly in the bug resolution?
IMHO, the only time this is a problem is when some pointy-haired boss decides that there’s a really important metric invovling bug resolutions. So now there are two problems.
[…] Resolved as Not Repro […]