Decisions around the product release – part 2 Martin Jansson
How is information handled around the decisions for the product release?
There are many different situations around this, the following is what I’ve experienced and my tips and tricks.
In many cases there is no information available or rather it is not presented to the decision makers. It also common that the information is not required to form the decision. The decision is formed based on gut feeling or something similar by the decision maker, but not on the information available from the project members.
In some cases information is produced based on IEEE standard test reports. It is the standard report that no decision maker has explicitly asked for it, where they get the standard information based on “best practices”. This report may contain many good things, but also many things that the test report writer should perhaps not spend time on. Consider how much you really want to spend on administration instead of testing. Michael Bolton and James Bach have lots of good stuff in their Rapid Testing material regarding this.
There are also controversies regarding measurements that you want to produce and report. Examples of such statistics could for instance be bugs (bug count, critical bugs, etc), how many tests you have executed, how many scripted tests, how many exploratory tests and so on. The list of possible measurements is almost infinite. When you are using metrics, ask yourself if this really give any valid information that can be used for a decision of some sort. If you use metrics, have it in an appendix and write your own conclusions and context around the metrics.
If you have a standard test report that you “must” use, ask the decision makers which information that you can get rid off and what they want to see there instead. A tailor made report for the decision maker is much more appreciated than a generic or standard report.
If you start from scratch with gathering information for decision making, ask them what information they would like to be able to a make sound decisions. Naturally each decision maker might need different information. All information should not be produced by the testers, which is sometimes the case. It is better that information comes from many sources: unit test coverage from developers, project management details from the project manager, test results and coverage from testers and so on.
There is a controversy around reporting your gut feeling as part of the information for the product release. In a sense I agree that it is good to avoid feelings in your report, only focus on facts. Still, if you write about your gut feeling as one source of information it should be ok. Gut feeling might tell something that you have a hard time explaining, something you cannot point at directly.
Test information for the release decision is difficult since different stakeholders want, and need, different things. Most of them want a lot of numbers, maybe because they can let the numbers make the decision? But you need a lot of knowledge about details in order to make a good analysis of many measurements, details that are difficult to give in a fast manner.
I wonder if it would work with a poll for all participants of a project:
1) Is the product ready for release?
2) If not, why?
Of course there will be some No answers, and the details for those can be weighed against “product opportunities”.
But in order to have effect, the information must often be given a long time before release. So in that respect, the release test report is the least important one?
Another way to look at status report is as a tool for the writers to make sure the most important things have been performed.
A poll would be good. Either if it is a yes or no they must state why they think so.
When it comes down to the actual release decision perhaps the question should be “Based on the information we have received and what we have discussed, can we take the risk to release this product now or do we need further investigation?”
The more I think of it the more I like the thought of having a poll like you suggest, Rikard.
This way you can substantialize gut feelings and put some real data or info behind them. And you will also get info from several project participants with different objectives.
This might also be good in order to let developers/programmers have their saying without having to care about their sometimes stereotype role(*) that wants “to release rather than stopping the release”. They now have a chance to “care about quality” as they should (and can).
The poll could though be unsuccessful due to that people are afraid of telling the truth or afraid of mention something that might be sensitive.
Then there is an option of having it anonymously.
The thing is to get the info, and that decisions can be made due to that more information is available that could influence a decision.
I need to try this out in the next release i am involved in!
(*) A role that testers often put on them as they assume that testers are the only ones who really cares for quality. Read more about such assumptions in (“The Ongoing Revolution in Software Testing”).