Scripted Testing: Filling out templates Henrik Emilsson
I saw an interesting interview with Rob Sabourin today http://www.youtube.com/watch?v=HZRXdaN7gkY (Thanks for the tip, Jon Bach!)
One thing he says in this video is: “… There are a lot of template junkies out there. [Testers are] filling out templates and not actually testing. That frustrates testers…”
Hey, isn’t this the same thing that happens in strictly scripted testing environments?
When testers are following detailed test scripts they are actually ending up filling out templates. With the same frustration as when filling out templates where you also don’t have to bother use your brain.
The result is the same for scripted testing as with a filled out template. Little mind work has been involved and so the information is very narrow and/or shallow…
It is sometimes said that templates are good because it is prevent someone from doing mistakes and/or making sure that the format is the same all the time. But isn’t a template to a document the same as what a script is to a test? A prescribed recipe on how to complete the task. I.e. it is about management control.
Isn’t it so that you easily can be hit by inattentional blindness as you fill out the template and only worrying about not screwing up the format? As with scripted testing, the focus lies in the format rather in the content. Both in scripted testing and in filling out templates you worry about making sure all things are filled out as expected, not the actual testing or the actual writing of informative text.
And, as with scripted tests, if we could fill out the templates automatically, we wouldn’t need to do them manually.
Why does every argument against scripting insist that scripts are written in stone? A template, a checklist, a script, a test case and so forth are merely frameworks. They are living things that develop as knowledge and understanding of the software, the environment and the user(s)’ needs develop.
I would be equally concerned about an employee that doesn’t/can’t understand the value and intent of a process, methodology, approach, school as I would about one who blindly follows the same one on every project without giving consideration as to why they’re using them.
Is there even such a thing as a purely scripted shop? The implication there is that your testers aren’t observing anything and simply executing what is set before them on paper. That’s a bad tester, not a bad methodology. If you have an exploratory tester with a pre-conceived set of heuristics in their head that they use every time, is that not scripted?
What about checklists? Are they not templates?
Hi CP,
Firstly, I forgot to insert the standard phrase: “Of course it all depends on where you are on the Scripted vs. Exploratory continuum”. 🙂
Update: (But I did wrote that this was regarding “…strictly scripted testing environments…” so I did leave the continuum-disclaimer out on purpose.)
I am talking about a purely scripted shop. They do exist, I can promise you.
And sometimes scripts are written in stone.
And I agree that this is bad testing, but it is not the testers that chose the approach; so we cannot call them bad. It is just their activity that is bad and they are sometimes forced to this.
I have been working in projects where bugs are invalid if they aren’t strictly connected to a test case, which in turn must be strictly connected to a requirement. This environment is not premiering Testing as I see it; it is premiering following exactly what the process says. It is robot dancing.
It made me start to guerilla-test and try to convince some stakeholders, but I still had nothing for that. So eventually I stopped doing that also.
I don’t see checklists per se as a problem, it is like you describe: it matters how you use it.
During many years I struggled with test plan templates that didn’t make sense and I didn’t see the point of what I was supposed to fill out. It was a lot of copy paste from last year’s test plan.
But one day I started using the checklist in chapter 11 of “How to Evolve a Context-Driven Test Plan” in the book “Lessons Learned in Software Testing” (Kaner, Bach & Pettichord). And suddenly the test plan made sense, not only to me but also to my project stakeholders. This was because it made me think about the content; what mattered; and what should be left out.
So using a checklist as a support in creating something is inspiring. Using a checklist just to make sure that you do exactly as it says in the checklist might be more of a dull thing.
I have also been in semi-scripted environments similar to what you describe, and it has been very useful.
When I got reminded today about the template junkies, I came think of how much of this behavior stems from organizations want to control everything and create a factory-like environment. For me, the connection between Templates and Scripted Tests became clear.
At the time I was struggling with test plan templates, I was fooling myself in thinking that a template is a tool that can help me. But it wasn’t. It was rather an obstacle to me. It was rather a tool for management.
I have seen contexts where this template or test script is necessary in order to use the feature correctly.
For instance, when handling testing around static IP, setting up routes and so on you need to know what to do to set it up correctly for the whole feature to work. So sometimes you need templates or scripts so that you are testing the right thing and not being stuck in a setup which is incorrect.
@Martin, hence the analogy.
“When I got reminded today about the template junkies, I came think of how much of this behavior stems from organizations want to control everything and create a factory-like environment.”
This is a key point. I’ve worked on a few large billing systems that spanned several companies and multiple teams. Controls were important to us to manage because a small undocumented change in one small program could end up causing integration issues in several other applications then everyone would spend time tracking it down, costing a lot of money.
Unfortunately, you get a manager on a project like that who tries to apply the same methodologies on smaller projects and it’s not always a perfect fit.
We do use scripts to some degree for acceptance testing, but we try to encourage our testers to only document to the level of detail that makes sense. When the cost and time of maintaining your documentation replaces effective testing, it’s time to consider another model.