Is our time estimation on testing valid? Martin Jansson

What do we actually base our time estimations on when delivering a plan to a project manager?

I know that we initially can have a vague idea on what to include and what must be done. I am sure that we can even make a rough estimation on how many resources we need in some cases. If we are testing something new, where we do not know the developers and the full extent of what is delivered, how do we know how much time and resources we need for testing? I have seen many plans that give an estimate, but how accurate can they be?

Once I did a resource plan for a year long test project, I was allowed to change the plan incrementally. I estimated that we needed quite many testers since the deliverables had been delayed and the original plan with incremental builds early did not work out as planned. In the test team we had a few disagreements about how many we actually needed, some thought that we were enough and some (including me) thought we needed more. We got 10% of the resources that I had wanted, but we managed somehow (to some extent). Decision makers seemed to be satisfied with the quality and the result of these tests. One of the reason for us succeeding was that we reported more bugs than the developers were able to fix. If we would have been more testers we would have found many more bugs for sure, but they would probably just have been postponed. The most critical ones were always fixed, but all others below that could not be fixed naturally.

So… should we when estimating resources and time focus on getting enough and the right resources just to keep the spot light of the resource discussion somewhere else? Lets say that bugs are fixed faster than we find new ones, will that mean that we are ready to deliver or that we need more testers to find even more bugs?

I think resource planning and time estimations is very hard. Michael Bolton has expressed, in an excellent way, many of the thoughts that I have tangled with. You can find his reasoning here:
http://www.developsense.com/2009/11/why-is-testing-taking-so-long-part-1.html
http://www.developsense.com/2009/11/what-does-testing-take-so-long-part-2.html

Combine the idea of having estimated how long time testing needs with an answer on how far we have come and especially pinning it down in percentage. Can we say anything truthfully here? Is it worth the cost it takes in planning and administration to give a accurate picture?

4 Comments
Michael Bolton December 6th, 2009

I remember sitting in on a conversation between a colleague of mine (a test manager’; let’s call him Alan) and Jerry Weinberg. Alan was expressing frustration at not knowing how to estimate when testing would be done, but being pressured by the project manager to provide an estimate anyway. Jerry pointed out that it wasn’t Alan’s job to decide when testing was or wasn’t done, since it wasn’t Alan’s job to decide whether there was enough information to ship. “But he wants me to provide an estimate anyway!” said Alan. “You don’t have to provide one,” said Jerry. “Go to his boss, or your boss (which turned out to be the same guy) and have the boss tell the guy to stop harrassing you and start managing.”

It’s not the testing department’s job to decide when testing is done, any more than it’s the waiter’s job to decide whether the diner is full.

—Michael B.

—Michael B.

Rikard Edgren December 7th, 2009

I agree that it is very difficult to estimate time; especially if anyone believes that the estimates will be true.
On the other hand, you can often make an educated guess (if you know/guess quite a lot about the project):
“With X persons for X weeks I think we can test the product pretty good, and give information that indicates the risks associated with releasing the product.”

Some factors that effect your educated guess are listed at http://www.gerrardconsulting.com/index.php?q=node/499 (What is the best ratio of testers to developers in an agile team?)

Martin Jansson December 7th, 2009

If I were employed I would do as you say Michael, but as a consultant, sub-contractor or similar its not that easy to take that course. You can try to make it go towards that perspective of testing but it takes time to challenge the view of testing within the limit of a project. Still, I think we underestimate what we can change and what changes we can affect with our own perspectives.

If you work with the same people, same kind of product, in similar environment, with same technologies but with new features in this environment I agree that you can make an educated guess.

Pauls list can be good to help consider test time.

What is even worse is when you get a whole new team of testers, where you have no idea of their skill. You do not know if they can work together, if they are good or bad etc. Are you also limited in how you work? Are you allowed to use pair-wise testing or are you forced to give out a subset of test cases to individuals who should focus on their task on their own?

The more I think about it the harder it is to give a number and a percentage.

Henrik Emilsson December 8th, 2009

Estimate time for testing is hard…

Sometimes, it is like if you conducted a murder investigation and were to estimate time on how much time you need.

Of course skill, experience and being familiar with product and context matters a lot, but as Martin also pointed out, being a consultant or an outsourced resource you lack two out of three important factors.