Scripted testing vs exploratory testing approach

I agree with the idea of a polarization between the scripted test approach and exploratory test approach. These approaches include how you perceive testing and a tester. Almost in the same sentence, some say that you do a bit of both scripted and exploratory testing. The perception on testing and how you conduct testing are two different things. I believe that by discussing them together and especially with the polarization example, it more than often confuse the listener.

Instead of saying that you do both scripted and exploratory testing, I think it is more fruitful to talk about testing and checking. M. Bolton and James Bach states that “A check is a component of a confirmatory approach to testing.” in the blog article Elements of Testing and Checking [1]. We might even be so bold to say that it is the main component, but then we might say that another part is smaller and that might not really be true.

The Checklist

One aspect of checks could be something similar to what the co-pilot does when assisting the pilot. If we see the pilot as the explorer, reacting to input and performing based on the current context. In certain situations the co-pilot brings up checklists to help go through a situation that is too complex to remember all the details about. It is up to the pilot and co-pilot to know when the checks are relevant or not. The same situation can be found at the operating table where you have several types assisting personnel that help perform a complex surgery. The use of a checklist in those cases would help avoid the easiest mistakes or misunderstandings. See The Checklist Manifesto [2] for more ideas for testing.

How does this apply to testing then? When we do pair testing there is usually one driver while the other is a bit more passive and documents. What if the driver is the pilot/explorer and the other is the co-tester handling documentation, checklists and checks? When doing collaborative test planning, group exploratory testing or other collaborative test activities, do we then split up in roles such as tester/explorer, co-tester, checker etc to help us define what we do and focus on? Does it provide value to do so?

Types of Checks

Here is a definition of a check [1]:

a check itself has three elements:
1) It involves an observation.
2) The observation is linked to a decision rule.
3) Both the observation and the decision rule can be performed without sapience (that is, without a human brain).

But could we split checks in different parts? What types of checks are there? M. Bolton describes questions that has to do with confirmation, verification, etc. Could we also break down checks into other categories such as checklists, test/quality patterns, guided checks and one-liner checks.

A checklist is a list of items that help you through a complex situation. The items in the checklist could be checks that need to be confirmed or verified to a certain extent.
A test/quality pattern is something that repeatedly gives suspicion that it might be broken so that you need to check it.
A guided check is what we previously called a scripted test. A set of steps that end with something that you wish to verify or confirm, something with a binary answer.
A one-liner check or a check idea is a brief statement of something that should be checked in a certain context that would give a binary answer, such as true/false, ok/nok or yes/no.

Checking clarifies an aspect of what testing is or is not, do these sub-categories help us clarify what we do with the checks? Could we find more types of checks?

Test Management Tools

Based on the assumption that there are at least two types of questions and at least two different types of answers, how do we structure and manage these today? The traditional management tools for testing can often handle all the above categories for checks, but they are unable to handle the answers from testing.

But what do we mean when we say handle in this case? Well, you can structure the checks, plan when you do them and report the result of them. You can then make reports on the overall progress and the overall quality of the system. You can also state what build, system or setup you used in the test. But since the tools cannot really handle the testing part, the progress and the picture on quality really becomes obsolete.

If we look at tools and techniques that are more aligned with testing such as SBTM, they are good at handling the testing story. They are not as good at handling checks, or rather the tools I have seen so far.

Can we and do we want to handle the result from our testing and checking in the same system? When working in small teams where there are fewer stakeholders to work for, the need for information sharing in a big system can be less important. But if you are working in an organization where you have teams in several countries and where the overall development organization can be thousands of people, it can be more important to share information in a bigger system. Still, are you actually able to share information to that many people in an efficient way just because you use an advanced management system? When a lot of people are involved in creating and sharing information there is also a bigger chance that the actual meaning is lost. You then oversimplify the situation that information sharing is easy. If the organisation is big, there is a big chance that the context changes over the organisation and that the information changes meaning over the organisation.

I’ve seen so many test management tools that try to solve the whole test process. By focusing on all aspects they also become quite crappy at all aspects. Picking the tools that are excellent at solving one problem can then be a lot more efficient even if you get to work with several tools. When working with test coverage, communicating your test ideas, reporting status, reporting progress and showing details on what you tested there is a big chance that one tool really cannot solve this for you. We know that we find new techniques from other disciplines that can solve a problem for us when testing. So why do many limit themselves to test case management systems?

Summary

Michael Bolton and many other great minds have explored and delved into this concept, I think we can find more pieces that we can shed some light on. I’ve identified a few areas and there are more to come.

References

[1] Elements of Testing and Checking – http://www.developsense.com/blog/2009/09/elements-of-testing-and-checking/
[2] The Checklist Manifesto – How to Get Things Right - http://www.amazon.com/The-Checklist-Manifesto-Things-Right/dp/0805091742

I just want to promote a really good blog post written by Michael Bolton where he describes the difference between Testing and Checking:
http://www.developsense.com/2009/08/testing-vs-checking.html

I wish that many managers, testers and developers read this post…

Cheers,
Henrik

This years EuroSTAR took place in den Haag, a city that had quite some rain, but also beautiful autumn leaves, and big churches.
The theme of the conference was “The Future of Software Testing”, and a recurring image was a traffic sign informing that the future is next turn to the right. I always thought the future was right ahead of us.

James Whittaker started with a really good and entertaining keynote where he argued that software is extremely important, and that the standard software testing will disappear (see http://blogs.msdn.com/james_whittaker/ for details.) On the other hand, he didn’t mention users or value (as Michael Bolton pointed out at the next sessions (sic!) questions), and he didn’t mention that testers do more than just verify requirements on different environments. We perform low-level and high-level validation and verification simultaneously.

Next I saw Derk-Jan de Grood who presented a mix of Exploratory Testing and structured process-focused testing. It didn’t feel revolutionary, since we Swedes have known about both these concepts for quite some time now.
The Austrian Strasser was a pleasant surprise. He told about Borland’s route from V-Model to Agile, and he stressed the people aspect many times. E.g. the teams themselves could decide if they would be split in smaller groups, and how. They have come a long way, and has a lot of unit test collaboration between developers and testers. “1 + 1 = 3″.
Tuesday ended with a keynote from the Google guys behind Testing On the Toilet (http://googletesting.blogspot.com/2007/01/introducing-testing-on-toilet.html) who has a really good story to tell. And it is a lot bigger than the Toilet, they have done a lot of things for more than three years in order to raise the awareness about testing at their company. A big accomplishment that I think is difficult to repeat at companies that don’t have “20% Time” (At twenty percent of your working time on Google you can do something completely outside your normal activities.)
After the presentation they were asked: “Do you have any objective metrics that can show your results?”
And I was really happy about the answer: “Metrics are difficult, there are so many factors involved. We rely on anecdotal information.”
They also stated: “There will never be a time when you don’t need a solid QA group.”
Wednesday started with Randall Rice sharing his thoughts on the trends that may shape the near future. Some general and good information, but not so much to write home about.
He recommended howtosplitanatom.com for articles on Web 2.0/3.0 and he said that a drawback of SaaS (Software as a Service) is that customers won’t have any control over which version they are running.
Julian Bensaid talked about outsourcing, and how to avoid, and quickly get out of “The Damage Zone”, the phase when thing are going a bit too bad for too long. He stressed “One Team” and said that “You have to innovate to make outsourcing successful.” “Devil’s Square”: quality, time, functionality, money. At question time, he let the audience answer their questions!

Then it was time for Michael Bolton’s double session workshop on Heuristics, which was the next best event for me at the whole conference. He really knows how to talk, he has important things to say, and the exercises stimulated good conversations with the other attendants. (The results are available at http://www.flickr.com/photos/23076491@N02/sets/72157608997920373)
Also, a good presentation stimulates your thinking to new ideas; and I thought of a new definition: “A good tester knows when to break the rules” and I’m wondering if some very good heuristics could be created at your company, where you have the details.
“Maybe testing isn’t supposed to be easy”, “We have to look complexity in the eye”, and he recommended positivedeviance.org.

Next it was my own presentation: Testing is an Island – A Software Testing Dystopia(link!). It went really well, except for one moment when I just couldn’t find any words. I excused myself, took a sip of water, and just started talking again; and after that I really can’t remember exactly what I said. I remember I said the important things, that I referenced some earlier presentations in the way I intended, but I can’t remember any details. I’m pretty sure I explained something in a completely new way that was really good, but I don’t know what it was. I see this as a good thing; my mind was in a state of flow.
Of course you only hear from the positive ones, but it seemed like people agreed that the human element is very important for us testers. I was also happy to hear that my presentation was very different from the other sessions, and that they like this different, thought-provoking angle.
I guess the introduction movie Mårten Ivert and Henrik Emilsson did was very helpful, just because it is so good!
One question afterwards was if/how I could convince people who think that emotions is a bad thing when testing. I said something about not having any proof, but that I might try by saying that we like emotions in life; so why shouldn’t we have them at work as well (Work is a part of our life.) Testers should also in some sense mimic users, that has feelings. I didn’t think about referencing Boltons excellent lightning talk on Emotions (http://www.developsense.com/2007/05/lightning-talk-on-emotions-and-oracles.html)
There was also a questioning regarding Agile development contradicting my dystopia; wasn’t that a way towards testers being more integrated in software development. I agreed, but also said that if the testers are brought in, in order to create unit tests instead of the developers, maybe it’s not totally positive. We need to look at many aspects.
Another good input was that testers who changes projects and companies, get a more holistic view, since they see more different things.
That’s a valid objection, and my defense was the team aspect, which is lost if people switch companies often. Best of both worlds might be to spend some time at a different place, and then come back.
Questions are the best part of presenting your ideas.

After this I went to a “Manifest Workshop”: Can the Past tell us the Future? This was nice because you got to spend some time talking to people about what has happened with our profession over the years. There are a lot of agreement and good ideas, but you also hear some things you can’t agree with, e.g. “Testers should have the mandate to change”, “Testers must have a special education with a title” etc.
This Manifesto extended over several workshops, and will be available at www.softwaretestingmanifesto.org; but I didn’t sign the acknowledgement list, even though they had an idea of “Anti-Manifesto” statements.
Wednesday ended with a casual session that generated some laughs: The EuroSTAR Testing Quiz.
Erik van Veenendal started Thursday with a keynote on TMMi. This is not my favorite subject, since “process-only” doesn’t say anything about how good the content is.
So I was glad that Erik started with saying that process isn’t enough; you need skilled people and a good infrastructure as well.
I also liked his statement that if you can take a Test Policy and apply it on a similar company; then the Policy isn’t worth anything. 
To succeed with a Process Improvement you need to have Management on board; and unfortunately they only listen to numbers (how long will that be true?)
You also need to spend a lot more time than Friday afternoons (Erik said 60% was necessary); and you need to focus on why you are doing it; what isn’t working good enough?
But Erik sometimes goes back to the Dutch formalized approach: the testing lifecycle should be Test Plan -> Test Design -> Test Script -> Execution.
“People want to change, but they don’t want to be changed.”
It was a good presentation, but unfortunately Test Process Improvement schemes aren’t generally good; most of all because it lacks aspects like creativity, and the ability to find all important errors.
I didn’t ask if there is a risk of becoming too mature, over-ripe, rotten.
On the other hand; you can read TPI, TMMi etc. and get really good ideas on what to do.

Egbert Bouman had presented his five favorite inspirational management Gurus, and how they could be applied to testing.
E.g. “Make company and personal interests run in parallel”, “People are complex.”
But I think it might be dangerous to transfer ideas from different areas without critical thinking. For instance, Collins Hedgehog Concept is about a company in the market that stick to doing what they are great at.
But in testing, we aren’t market-driven, we rather provide services/information to the project; and wouldn’t it be a bad idea to test software in the same way year after year?
He agreed a bit on this at the Gala Award Dinner, but also said that it differs if you are employed by a company, or if you are a consultant. But we discovered that we both were fans of Kierkegaard, and Morten Hougaard joined in this appraisal. “The truth is the subjectivity.”

I skipped Stuart Reid’s workshop on ISO29119. I was interested in how they are working, but I was sure I wouldn’t like the result; too much focus on standardization is killing creativity.
Tim Koomen jumped in at last minute and gave an overview of the testing industry in the past, present and future. He has a good understanding, and talked a bit about Pairwise (I don’t like it), Exploratory (I like it) and Model-based Testing (Surely good in the very few situations it is truly applicable.)
He also had survey results showing that only 1/3 of software testers are using Test Design Techniques (on the other hand; who isn’t implicitly using Equivalence Partitioning all the time?)
“Let’s use the existing techniques a little bit more.”
It was also nice that he brought up the classic Triangle Exercise.

John Kent have created a better Entity Model for Software Testing. “All models are wrong, but some models are useful.”
He used the name Test Condition for Test Ideas, which I think are central in our universe.
He showed a Test Case example, and I really feel sorry for all those testers that has to write or follow Expected Results for each and every test step in a Test Script.
He had a Real World Waterfall Model that started with “Produce requirements which are incomplete and slightly wrong.”
And he ended with the idea that in some situations you might better throw out these Entities and go for only Test Ideas; the tester will know what to do.

After lunch break Michael Bolton gave probably the best testing presentation I have ever seen: Two Futures of Software Testing.
He was wearing horns when talking about the Dark Future that is static and mechanic, and a halo when talking about the Light Future that has the testers human skills in the centre.
His message is very close to my own, with the difference that he has a lot more to say, and he does it extremely well.
It was so good, so I didn’t have time to write down any of all the good things he said, an old version of the presentation is available at http://www.developsense.com/presentations/twofuturesofsoftwaretesting.pdf.

The last keynote was James Lyndsay: Becoming Agile – Reshaping Testing for an Agile Team.
Agile exists, it is growing, and often succeeds with its projects, independent of the inclusion of testers or not!
But the testing performed is confirmatory, and that isn’t enough, so the tester approach is needed.
And in some ways; Agility and Testing is conflicting. Agile is highly optimistic, and testing is perceived as the opposite. Testing is used to be independent, must now be part of a team commitment. To explore for dangerous information might need feel good.
It is not necessary for a tester to know programming to enter an Agile team, but it is an invaluable asset.
He showed some statistics from www.ambysoft.com/surveys/agileFebruary2008.html.
And his main point in a very good presentation was that testers should help the team learn more about what they are building.

The conference was concluded with dinner in Grote Kerk where I sat next to a penetration tester; and beers at a pub, which really encourages many conversations.

Top 3 themes on the conference as I saw it was: People, Complexity and Agile.
EuroSTAR is a good conference, it has a nice aura of knowledge sharing.
Other things: Hoegaarden in Holland has so much more aroma than in Sweden.
Stuart Reid was walking around with a yo-yo.
My main attraction on the EXPO was the company that had a chess board. I played a game, won, and was thereby included in a lottery; and since only one other person won, I had the 50% luck of actually winning an IPhone!
I didn’t have the chance to see: Mieke Gevers, my nice Track Chair; The good title “Imagination is more important than Knowledge”; Fabian Scarano who in his paper and presentation mention serendipity.
It was a tie between really good Mexican and Argentinian dinner.
Isn’t it true that good testing should create low-hanging fruit?

Next Year EuroSTAR will be held in Stockholm, again! A bit boring; Barcelona seems like a more exotic place at least for me.
My current idea is to create a presentation consisting only of images; have a good name like “7 Ways to Explain Software Testing”, and then write a fluffy and nice abstract…
Or maybe try titles like Test Ideas – the Center of the testing universe; Generic Test Ideas for Data Types; What I Learned After 9 Years At Spotfire; The Holistic-Subjective School of Testing; The Eye of a Skilled Software Tester; Verification, Validation, and Everything In Between.
/Rikard