What is essence of the discussion on release team/showstopper meetings?

I am assuming that there is a meeting. I’ve been to many different kinds of showstopper meetings and most companies handle them differently. One important item on the agenda for the meeting is usually the bugs that are found late in the project, thus at the end phase where they might be considered showstoppers.

When these bugs are discussed if they are to be fixed or not, I’ve seen cases when the argument is a bit vague. The actual topic is sometimes not brought to the surface, instead it is someone who “feels” that this should be fixed or it should not. What is it that they feel? Articulate those feelings and try to be clear on what it is you see. Why should it be fixed or why shouldn’t it?

In these cases I think it is best to be clear that we talk about cost, at least in most cases, in one sense or another, thus the cost of not fixing it or the cost of fixing it. Some costs will affect support, others will affect marketing and sales and some will affect development itself. I can recommend trying to get everyone attending the meeting to focus their thoughts on what the bugs will cost you. Is a few days slip perhaps worth it? In some cases the date for the release is extremely important, this is most often the case for bigger organisations. Instead it is easier to make some changes after the release decision and make a few patches afterwards. The actual release will not reach the customer the same day anyhow. Either way it is cost you discuss and you most often decide based upon it.

Cem Kaner has written an article on this that I’ve used many times when setting up these meetings. You can find the article here.

How is information handled around the decisions for the product release?

There are many different situations around this, the following is what I’ve experienced and my tips and tricks.

In many cases there is no information available or rather it is not presented to the decision makers. It also common that the information is not required to form the decision. The decision is formed based on gut feeling or something similar by the decision maker, but not on the information available from the project members.

In some cases information is produced based on IEEE standard test reports. It is the standard report that no decision maker has explicitly asked for it, where they get the standard information based on “best practices”. This report may contain many good things, but also many things that the test report writer should perhaps not spend time on. Consider how much you really want to spend on administration instead of testing. Michael Bolton and James Bach have lots of good stuff in their Rapid Testing material regarding this.

There are also controversies regarding measurements that you want to produce and report. Examples of such statistics could for instance be bugs (bug count, critical bugs, etc), how many tests you have executed, how many scripted tests, how many exploratory tests and so on. The list of possible measurements is almost infinite. When you are using metrics, ask yourself if this really give any valid information that can be used for a decision of some sort. If you use metrics, have it in an appendix and write your own conclusions and context around the metrics.

If you have a standard test report that you “must” use, ask the decision makers which information that you can get rid off and what they want to see there instead. A tailor made report for the decision maker is much more appreciated than a generic or standard report.

If you start from scratch with gathering information for decision making, ask them what information they would like to be able to a make sound decisions. Naturally each decision maker might need different information. All information should not be produced by the testers, which is sometimes the case. It is better that information comes from many sources: unit test coverage from developers, project management details from the project manager, test results and coverage from testers and so on.

There is a controversy around reporting your gut feeling as part of the information for the product release. In a sense I agree that it is good to avoid feelings in your report, only focus on facts. Still, if you write about your gut feeling as one source of information it should be ok. Gut feeling might tell something that you have a hard time explaining, something you cannot point at directly.

Who makes the decision that the product is ready for release?

There are many different cases of this situation, here are a few examples.

In some companies it is the QA department that makes this decision. This means that it is QA that takes the risk for the release. It also means that if something happens after the release it is QA that is to blame for making a “faulty” decision. I’ve been in situation when management said “Why did they not test that?” or “How could they miss that area? It was so obvious.”. If this happens QA will be worried more about what the rest of the company will say and will in some situations say that the product is not ready for release just to avoid possible blame. Does QA have the authority to get longer time to test, are they able to get more bugs fixed, what authority does QA have?

In some cases it is the project itself that makes the decision where the whole group (developers, testers, project manager and so on) decides. This means that the project and most often the project manager him/herself takes the risk for the release. So, the blame will fall on the project and its members. Is the project able to get resources, time etc if need be? In most cases no.

In other cases it is management or a release team consisting of managers from all departments that makes the decision. This means that someone outside of the project makes the decision for the release. It also means the blame will fall on this team, not the project itself. It is also common that this team can affect resources, time etc for the project.

We also have the absurd case where no one makes the decision. The release just happens and no formal decision was made. In this case it is more of a bad process issue that this happens. Management is responsible unless the project manager has the formal responsibility in these situations.

If you are unsure who makes this decision you really need to discuss this with management. Someone must take the risk and responsibility for releasing the product. I recommend that it is management or a release team that makes this decision, since they have the authority to make changes. This naturally leads into the role of QA, are they Quality Assurance or Quality Assistance (as Cem Kaner so nicely put it)? I think that if testers can avoid making this decision it will make their life a lot easier.