Utopic estimations in testing Martin Jansson
Making estimates on a test assignment is hard. What you include and exclude varies between each person. As I see it, there are many things to consider that might make the whole test effort longer and make it more complex. So, instead of guessing or giving a random number consider this…
When you are asked how much effort you need to spend on the various aspects of testing for a new feature it is quite common you picture yourself that …
- it is the person him-/herself who does the estimation that will perform the assignment/testing
- each assignment is done by just one person instead of the whole or part of the test team
- the test team have good cooperation with product managers, developers, other teams, other testers that are involved in the assignment, no obstacles at all here that will steal time
- all hardware is available and is fully usable from start
- all hardware will remain in perfect shape, nothing ever get broken, nothing ever need to be repaired
- all software is available and is in perfect shape
- test instruments, used in testing, are fully functional with latest revisions of hardware and software
- each tester in the test team has tip top domain knowledge and no education about the assignment is needed, there is nothing out there that is complex or hard to understand
- each tester in the test team is executing tests on the correct test object, no confusion about revisions or if parts of the test object is correct
- the computers in the test lab are so fast and powerful that everything goes smooth at all times, even the network is fast at all times
- performing each test goes fast without any faults found, everything works perfectly
- there is no need for assisting developers pinpoint bugs, since none will be found
- there is no need to verify bugs found, since none will be found
- there is no need for performing any kind of regression, we did the right thing from the start
- all documentation that is interesting for the testers is in such good state that nothing is unclear
- nothing outside the plan for the test team ever happens during the whole project, you are always on track on all things
- the test planning is so good that it is always on time without any delays
- there is no need for any meetings or discussions inside the test team or outside, administration of the assignment is not needed at all
- there are no conflicts in the test team, never have been and never will be
- the test team is constant in size
- no one will ever be sick, have kids, have to sleep, go to the toilet or have lunch breaks
- the test team is perfectly aligned in how they should work with testing
- the test team is perfectly aligned with how the team is managed, no confusion or downtime
- all test automation works perfectly at all times, no disinformation at all
- automated tests never break and always report pass
- automated tests does not need to be tested since the developers never make any mistakes
- the test team will not introduce any new tools or ways of working ever
- everyone in the test team estimate the same way and has the same idea on how it should be done
- personnel that is going to assist the test team also loves testing and will have no issues at all being removed from their previous position, it won’t affect their work at all
- personnel that is going to assist the test team will always bring joy and happiness to the current test team, there will be no conflicts ever
- utterly new personnel has full knowledge of corporate culture, where to get information, who does what and when, what is expected of them
- utterly new personel who stays for three weeks will be productive the whole time
- utterly new personnel never need anyone to mentor them or introduce them to assignments
… and you have not even gotten to the actual testing or test planning part.
I know… if you think like this will you ever get any assignment? Perhaps it is just better to give the amount of time that your stake holder want to hear? Still, I think that if you identify things that steal time from testing or make the assignment take a longer time, there is a big chance that management might be able to assist with removing these obstacles or see that you have thought this through.
Further reading related to test estimation:
http://blogs.msdn.com/testing123/archive/2007/04/11/demystifying-test-estimate.aspx
http://blogs.msdn.com/alanpa/archive/2007/03/13/test-guesstimation.aspx
http://www.developsense.com/blog/2009/11/why-is-testing-taking-so-long-part-1/
http://www.developsense.com/blog/2009/11/what-does-testing-take-so-long-part-2/
http://www.developsense.com/blog/2009/10/when-do-we-stop-testing-one-more-sure/
http://www.silverpath.com/resources/Silverpath-EstimatingTestEffortWhitepaper-080812.pdf
http://inderpsingh.blogspot.com/2010/03/how-to-estimate-testing-efforts-6.html
http://www.satisfice.com/blog/archives/124
and an infinite amount of good books on testing where most of them touch the test estimation subject.
Ouch, that’s a long list of things that can cost time.
I see estimates as a necessary evil, that you sometimes don’t have to do.
It’s easily kind of self-fulfilling, so you don’t do what is needed (could be faster, often slower), you rather do the best you can in the allocated time.
Testing is so dependent on so many things, and you often don’t know the best way to do the things until afterwards (after several different approaches.)
And sometimes also the journey there might yield important information; even though you didn’t accomplish your specific task, it might have been well-spent time.
A benevolent look at estimates says that it’s good to have something to start with, we pad the estimates (to cover your list) and do our best.
A cynical feeling regarding estimates come when the estimate is rejected and you do it once again…
In some cases I think it is good to identify things like the list above in order to see that it might not be so easy to do after all.
In other cases you might take the risk and instead just say that I will only spend 20 hours, unless a volcano errupts or something similar.
In either case I think it is important for other testers who might be affected by the estimate to understand what the specific number meant and what you included in it. If you wish to learn from the estimate in order to make better estimates in the future, it is good to articulate what you include and exclude.
An alternative usage of this good list is as checklist when identifying risks for the testing part of the project
one addition:
* you notice that the task isn’t covering the most important parts, so you need to re-estimate
Great thoughts Martin!
I think that this serves as a good reminder of that it is easy to present time estimates as if it was done by yourself in a perfect world. But as easy it is to miss this, it is as easy for the receiver to forget to ask for the risks.
And the hardest thing is to do a risk assessment of the time estimates.
The list you present is really a great checklist of possible risks that threaten the (utopic) time estimates; and you could go through this in order to validate your estimates (for each risk that indeed is valid in your project).