It’s a truth that we can’t find all bugs, but is it really a truth that we can’t find all important bugs?
And it’s a cliche when used as answer to the (sincere) “why didn’t you find that bug?” question.
Testers are paid to find important information about what they are testing, and included are the big bugs, so you can fix them before the software is used in production.

Besides some good things about forthcoming test strategies, I hope you also can say: “if you have read the test plan and status reports it shouldn’t be a surprise bugs like this got away.” Well-informed risk taking is part of our business.
However, if there is a test that seems important, and would take a couple of minutes to perform, this argument is not valid. Testers (in a serious project) should try to perform all important tests that can be performed fast.

There are testing lessons to be learned if the answer is:
* we didn’t think about that scenario
* it wasn’t included in the requirements
* someone told us not to test that
* we had no idea that the customers where running that kind of configuration
* we found it, but didn’t report it well enough
* we didn’t put the bug there, and they hid it awfully well
* this type of issue can’t be found with automated tests

And there is the occasional invalid question, e.g. when the bug was reported in a good way, or when there was no testing time alotted for a late change.

Bottom line is: take the question seriously, and try to improve your testing to avoid this and other important issues (next time it will probably be another kind of issue…)

As a user of Facebook I feel really helpless when nothing works as it should (as was the case with the latest GUI-update). Posts were stochastically shown in the feed and a lot of errors occurred in various situations. A lot (all?) of my friends on Facebook experienced the same problems.

When there are lots of bugs on flight booking sites I get so frustrated and angry because I cannot complete my task. E.g., the booking system for Ryanair (at least two years ago when I last used their booking system and never will use again).

But what can I do as a frustrated enduser?

The problem nowadays is that you as a user and consumer don’t know what to do with your complaints. The companies shows no respect as long as the users are hooked on the application and needs the solution to problem that the application provides. But they do not care to fix the bugs that might ranging from annoying to critical.
So the loser is you! And it becomes a rage against the machine because of the absence of real people to talk to. Or some people might think that they did something wrong when it really was lousy software that caused the problems.

But in fact all other users of these applications experience similar problems. So you as an enduser is not alone! This is something that hits many people on their own but who lacks a community i.e., if we were employees in a company using the software we would have protested.

What should we do!?

Should we create a site similar to http://www.badsoftware.com/ (Kaner & Pels) or http://www.badsoftware.com/alienwaresucks/ (Kaner) in order to channel some of the frustration and gather it into some powerful criticism?

Or are there other channels that we could utilize?

Do you have any ideas?

In my local food store they have this system where you scan the price tags on the food you buy and most often smoothly able to pay and exit without having to stay in any long queues.

A time back they must have changed software in these scanners because their behavior changed and bugginess increased. The funniest bug or feature, as they themselves would most certainly call it, is when you are finished. You then scan a finish code which then sends a signal to the scanner, then you are able to pay. When you perform this last scan you will now hear a loud beep from the device, previously this beep was used when there was an error of some kind. So, everyone (at least everyone who I have seen do this) perform the last scan and upon hearing the beep they exit the queue and go to the cashier. The cashier then tells it is supposed to sound like that and that it is perfectly normal. This happens every time for everyone, at least once for us technocrats… but prolly each time for those who are a bit scared of technology still. One of the main ideas with using this device is to minimize the effort for the cashier by letting customers check-out on their own. This sound stops that feature.

Another funny bug that have appeared quite recently is that it takes a bit longer to scan wares, I mean from less than a second to close to ten seconds per ware. Pulling up a box of tomato sauce where you must scan each of the 12 cans will now take about two minutes… you just stand there continuously pressing scan… waiting and building up on that hysterical laughter. The idea of Shop Express has lost a bit of its flavour, still when it works as expected it is indeed a lot better than using the normal queue.

Consider a building with a few broken windows. If the windows are not repaired, the tendency is for vandals to break a few more windows. Eventually, they may even break into the building, and if it’s unoccupied, perhaps become squatters or light fires inside.

Or consider a sidewalk. Some litter accumulates. Soon, more litter accumulates. Eventually, people even start leaving bags of trash from take-out restaurants there or breaking into cars.

based on an article titled “Broken Windows” by James Q. Wilson and George L. Kelling in March 1982. You can find a bit more about the background here.

It is my belief that the same thing happens with products and product development. When bugs start to accumulate in an area and where the bugs do not get fixed, you will as a tester lower your standard on what is a bug or not by comparing the new found ones to the vast amount that already exist. Eventually you might get the feeling that you might not even bother reporting bugs, because you know that they won’t get fixed anyhow.

I think this is mainly a managerial problem. Examples of these kinds of problems can be when…

• focus is on implementing new features, just getting them in there
• covering an area with tests is more important than actually finding bugs and getting them fixed
• the threshold for getting a bug fixed in the late stages of a project requires earth quakes or miracles

Many other testers talk about this phenomena as when products are going rotten or something similar.

How serious is this problem for testers? What do you think?

You are on a quite large company where there are several QA divisions, several layers of management, several listeners to each step of the development process. It is the final weeks of the release. You are about to enter a bug which seem serious but you are not sure. You can take at least two paths here…

Take the easy path and you report what you have seen so far and scribble down a few quick repro steps on what you think might be the bug. You do not dig deeper cause its been a long week and it is your right to go home now.

… or …

Take the harder path and you really try to hone down what is wrong. You try to find some repro steps on what it is. You try to see how often this bug appear and what impact you think it has. You extract log files, config files and all other information that you think might help the developer. You proof read the bug report and make sure you have gotten everything in there. You let someone else take a look before entering it.

As the submit of the bug takes place you see that there were many who got the bug in the mailboxes as a final delivery at the end of the week.

Chosing the easier path might have come to a situation where you as an observer…

• do not believe the bug
• ignore the bug because it seem too vague
• think the testers are crap and become unsure if their previous bugs were infact not correct as well
• are scared that the quality of the release is in danger because you have no trust in the bugs
• become angry because the testers do not take their job seriously and enter a bug in such bad state in such a critical moment
• become angry as a tester thinking that the other test team gives you a bad reputation

The list could go on for some time…

Chosing the harder path might have come to a situation where you as an observer…

• are thankful as a developer that you did not have to work the weekend to try to find the reprosteps
• are thankful as a project manager that you were able to make a bug classification more easily before everyone had left for the weekend
• are impressed as a manager that the testers give the extra effort to save so much time for others
• are thankful as a tester that the other testers take pride in their work and deliver at every step

The easy path is quicker while the harder path takes a bit longer time. Depending on your thouroughness and completeness the result will differ enourmously. Naturally this is not always the case, but I’ve seen this happen so many times.

This post was triggered by blog post Resolved as Not Repro - http://thetesteye.com/blog/2009/06/resolved-as-not-repro/
I believe that bug systems too often are used with onlý a this-project-right-now approach, where you care most about just getting all items dealt with.
This is perfectly fine for one-off type of projects, but does not work fully for software where the bug system is (one of) the most important sources regarding quality information.

Since I use a holistic-subjective approach to testing, I need to consider both now-and-then, and also me-and-everyone-else.
Testers, developers et.al will treat bugs differently if they are resolved As Designed, or Fixed, or External, or Worksforme, or Not Repro, or Invalid, or Postponed, or Won’t Fix or Duplicate. 
At the end of this project, or next version, I might look again at all bugs set as Not Repro, or re-verify the most important Fixed bugs.
If there are many By Design issues, some usability folks might look into the details, and consider new approaches for the area.
Support people might search for information, and act differently depending on the solution of the related bug.
And also people that think that metrics matter, might be even more deceived, if the resolutions aren’t the correct ones.

My point is, you can’t really know how the information may be used, so do your best.

Lets say that you have a bug system; and for each bug you have the two fields “State” and “Resolution” where the following values are valid:
State: New, Assigned, Resolved, Closed.
Resolution: Fixed, Invalid, Won’t fix, Duplicate, Not Repro.

Further, you have a field where a product version number should be entered; i.e., the earliest version number of the product where the bug can be reproduced in.

Now, lets say that you report a serious crash bug that is reproduced in the latest build of the product e.g., version 1.1.2. (Oh, I forgot to mention that you work as a software tester at a software company and this happens during development of the upcoming release of your product.)
Two weeks later the bug is resolved by the assigned developer; the State and Resolution is set to “Resolved – Not Repro” and the following comment is added by the developer in the bug:
“I cannot reproduce this anymore (code is same as in build 1.1.4). This might have been fixed when other bugs in the same area were fixed. Resolving as Not Repro…”

So what do you think?

Do you think that this bug should have been resolved as Not Repro?
If not, what resolution would you have chosen?
Would it make any difference if the resolution “Works for me” had been a valid resolution instead of “Not Repro” (as in Bugzilla)?
Any other thoughts on this?

The front door

The bug that a co-worker found was that you were able to pull open the door a quarter of an inch, just enough to be able to put in a little stick and then wave it infront of the radar. It actually goes quicker to use the stick instead of the card.

I confronted the installer of the security radar and told him about the bug. He answered me “No, that is not possible.”. I told him again, it is so and that one of our consultants use it since he has no pass card. He answered me again “No, that is not possible.”. I did not have time to argue so I left.

I am now going to notify the owner of the house so that they understand the problem. Apparently there are more people using this trick when they have forgotten their card, so the stick is just below the pass card holder for easy access. A few years ago there was something called “Not my job award!”, I guess this would fit into this category.

What is essence of the discussion on release team/showstopper meetings?

I am assuming that there is a meeting. I’ve been to many different kinds of showstopper meetings and most companies handle them differently. One important item on the agenda for the meeting is usually the bugs that are found late in the project, thus at the end phase where they might be considered showstoppers.

When these bugs are discussed if they are to be fixed or not, I’ve seen cases when the argument is a bit vague. The actual topic is sometimes not brought to the surface, instead it is someone who “feels” that this should be fixed or it should not. What is it that they feel? Articulate those feelings and try to be clear on what it is you see. Why should it be fixed or why shouldn’t it?

In these cases I think it is best to be clear that we talk about cost, at least in most cases, in one sense or another, thus the cost of not fixing it or the cost of fixing it. Some costs will affect support, others will affect marketing and sales and some will affect development itself. I can recommend trying to get everyone attending the meeting to focus their thoughts on what the bugs will cost you. Is a few days slip perhaps worth it? In some cases the date for the release is extremely important, this is most often the case for bigger organisations. Instead it is easier to make some changes after the release decision and make a few patches afterwards. The actual release will not reach the customer the same day anyhow. Either way it is cost you discuss and you most often decide based upon it.

Cem Kaner has written an article on this that I’ve used many times when setting up these meetings. You can find the article here.

It cannot be exactly defined what a bug is, or how it should be reported. And each tester, developer, project manager et.al. has her own way of writing, thinking about and handling bugs.
I like to think of this as taste.

Do you prefer having all details in a bug report, or only including what you think is important?
Would you rather put high priority or low for an in between case?
Can someone else recognize the style your bug reports?
Do you often think your bug reports are more important than others?
Are you often finding the same type of bugs?
Should the first letter in a title be capitalized?
How do you feel when your bug is returned As Designed?
Do you prefer informal reporting for some type of issues or persons?
Do you count your bugs, and if so, why?
Is the audience preferences for bug style more important than your own (efficiency)?
Which bug would you like to find?

Is your taste for bugs impeccable, in your context?