Let me begin by a quote from T.S. Eliot:

We shall not cease from exploration, and the end of all our exploring will be to arrive where we started and know the place for the first time.

One of the most important things with Exploratory Testing is that it allows for you to learn something during the exploration. It is urging you to think for yourself; draw conclusions; misunderstand; learn from mistakes; see the full picture; noticing details.
Learning about the product and the context is a key element in discovering problems that might bug users and stakeholders.

I have a problem with remembering and learning stuff if I (or someone else) have written something down and I’m using the text as the answer each time. E.g. a set of instructions on how to perform a special action; a login password; etc. But if I have done the thing myself, I have no difficulties in performing the special action. And perhaps most importantly, I have an understanding of every needed step and why each step is needed.
When I am asked to review something, it is important that I explore the area in order to learn something from it. To me, this is (almost) the only way to see what is missing. I can do a good job by finding problems in the things that are already in there, but in order to see what is missing I have to learn about the area and what it really means.

It is the same thing with Scripted Testing vs. Exploratory Testing. If tests are scripted, there is no need to reflect upon what you do. The script is there so you don’t have to think. An exploration, on the other hand, requires thought work. It requires that your brain is in an alert state and that you learn something from what you perceive. ***

I also think that the learning part of exploratory testing is what matters when comparing two testers with similar resumes. Ask the candidates if they have had an exploratory approach in the reference projects, and ask them what they have learned.
I bet that you will spot those who have learned something and thereby improved their testing skills, including how to think about the context and its affection. I also believe that they have improved their problem solving skills and boosted their creativity along the way.

Do you have any learning experiences that you can share with us?
Have you experienced when you didn’t learn something? What did that do to you?

*** Note: Of course it depends on where you are on the Scripted vs. Exploratory continuum.

I have written a small article for The Testing Planet about Testing & Creativity. (update: web article here)
It contains the potato from this blog, but also some new content on “cheating”.
Feel free to add comments here!

I think creativity still is too little recognized as an important aspect of our work.
We should promote it more, both because it is a vital aspect, but also to make the profession more interesting to the next generations.

There are (at least) two upcoming presentations on creativity in software testing:
Rob Lambert – Is excessive structure killing our creativity, blog, video, program.
Rikard Edgren – 5 Steg Till Kreativ Testning, SAST 15 år (a Swedish version 2 of EuroSTAR 2007 presentation Where Testing Creativity Grows)

There are a lot of other content in The Testing Planet paper, my favorite being a compelling story from Scott Barber on the publically available Black Box Software Testing course at testingeducation.org.

For quite some time I have felt that the classic test design techniques don’t add up to the needs of software testing that tries to find most of the important information.
At EuroSTAR 2009 it dawned on me that it is time to describe the method that I, and many, many others, have been using for a long time.
By talking more around this, I hope we can spread the usage of this test design, improve the way we do it, and make it more understandable for other people, avoiding the comment: “oh, you’re error guessing.”
What I will describe could be called a method, activity or a skill, but I like to think of it as a test design technique, that is seen as a tester’s most powerful weapon, much better than equivalence partitioning, decision trees etc.

Grounded Test Design: learn a lot, and synthesize important test ideas.

To explain this further: Software testers should follow the thoughts of social science Grounded Theory; we should use many sources of information regarding the subject (requirements, specifications, prototypes, code, bugs, error catalogs, support information, customer stories, user expectations, technology, tools, models, systems, quality objectives, quality attributes, testing techniques, test ideas, conversations with people, risks, possibilities); we should group things together, use our creativity, and create a theory consisting of a number of ideas capturing what is important to test.
Grounded Test Design is for those that want to test a lot more than the requirements, that understand the need to combine things of different types, to look at the whole and the details at the same time.
This might seem heavy, but people have phenomenal capacity.
It has a lot more to it than the closest technique error guessing; for instance, it is not only about errors, it can also be investigations, or questions regarding dubious behavior.
It isn’t guessing, it is a qualified synthesis of knowledge.
And also: it isn’t (and doesn’t sound like) something done lightly, something unstructured, or unprofessional.

Being a technique, it is something that can be applied in many different situations. The most informal Grounded Test Design might happen extremely fast in the expert tester’s head when thinking of a completely new idea while performing exploratory testing; or it could be the base for test scripts/cases/scenarios, that are carefully investigated and designed in advance. Good software testing uses many different methods.
A formal Grounded Test Design (to be defined, tried and evaluated…) might be best used by someone completely new to a product, or maybe most efficiently performed in diversified pairs.

Grounded Test Design has a loose scientific, theoretical base in Grounded Theory (Strauss/Corbin), used in social science as a qualitative, thorough analysis of a phenomenon.
The scientists examine the subject very carefully, gather a lot of material, document codes and concepts, combine them to categories, from which a theory is constructed.
There is no hypothesis to start with, as in the traditional natural science, and that’s quite similar to software testing; it would be dangerous if we thought we knew the important questions at once.
I think it is good to be slightly associated with this theory, especially since it emphasizes that the researcher must use her creativity.

On the other hand; do we really need this word, maybe we already have enough words about how to do good software testing?
Or is it a good word to have in reserve when explaining to managers why testing is difficult, takes a lot of time, and must include manual work?
Or should we go further with this and try to develop a framework for performing structured analysis of everything that is important for testing?

Opinions are very welcome.

When you come up with a new test idea, you are using your knowledge and experience, but there is also some sort of stimuli that triggers the idea. Something you see, hear, understand or think about.

You seldom think in totally new ways, you rather combine things in a new way.

These are my favorite test idea triggers:

* think about each feature (and connect with testing or technical knowledge)

* create or understand a model of the software

* think about interaction between items in the system

* thinking about quality attributes (CRUSSPIC STMPL + Accessibility)

* read bug report titles for similar functionality

* read test idea lists

* play around with the software, or a prototype, or a previous version

* change perspective, see your software as a little box among many others

What triggers you?

Do you get help from Session Tester Primers?
(http://sessiontester.openqa.org/download.html)

At EuroSTAR 2009 I will present “More and Better Test Ideas“; the main idea being that testers could generate many different types of test ideas, and communicate them in a condensed one-liner format.
If you have great tips on how to come up with really good test ideas, or want to review the paper I’m about to write, let me know.

/Rikard

I spent one day at Øredev 2008 (http://www.oredev.org) since they invited me to give the Where Testing Creativity Grows (http://www.thetesteye.com/papers/where_testing_creativity_grows.doc) presentation.
I arrived ten minutes after the start of James Bach’s keynote The Renaissance Thinker, where he argued that 1972 (Chapel Hill) ruined good software testing.
People started focusing too much on templates, processes, best practices; people forgot about people.
But things are getting better since the Exploratory Testing has been coined and used with more respect.
We also have Agile that is using cyclic learning, in opposite to Waterfall and V-Model where no learning is modeled.
He likes it if people say “James, maybe you are one of the mystics?” and explained his self-learning style of Buccaneer-Scholar.
“Software Engineering is primarily social.”
He has a lot of good things to say; and he presents them in a capturing manner.

This conference is nice since it includes developers, testers, project mangers et.al.
But I stayed at the Test Track all day; starting with Isabel Evans, who is interested in processes and people; and stresses that it is the people in the team, that needs to define the process.
Quality is difficult, since it has so many attributes; the customers view being the most important.
Isabel is criticizing the “standards” from the inside, but think it is important to measure things and report back, e.g. by showing how much money is saved by doing software testing.
“Agile works since it is cross-functional; we work together.”
She also broadened the spectra of our risk thinking; which too often focus on internal risks. But there are Contractual or Regulatory Risks as well as Social Ethical Risks. It is good to think bigger.

Then it was time for my own presentation with about 60 attendees. I had an interesting start with technical problems as my machine wouldn’t use the projector after I launched a movie inside PowerPoint (Yes, I use Vista.)
So I had to improvise a bit and focus on the audience instead of my slides.
After this the presentation went well; and I was happy that an attendant could tell about their paired testing (both scripted and exploratory) where they found three times as many bugs (plus the learning!)
A good question after the presentation was “When do you know you have used enough creativity?” with the answer “Well, you don’t…”

After a lunch break Kevlin Henney talked about “Know Your Units”, with the clarification that a unit test that interacts with something (file system, database, network) isn’t a real unit test.
Programmers are responsible for unit testing (“The programmer who wrote the code is in the best position to test it”) and ”QA” for system testing. (He didn’t mention any collaboration, which I think might be the best.)
POUTing -Plain Ol´ Unit Testing – is passive.
TDD – Test-Driven Development – is active.
DDT – Defect-Driven Testing – is reactive.
And sometimes we can settle with GUTs – Good Unit Tests.
He recommends a black-box perspective for unit tests, where you focus on behaviors, or even requirements. White-box testing can end up testing that the code does what it does.
“We say that system testing takes one month, but in reality, the testing takes a few days, the rest is spent fixing bugs.”
“Testing is a way of showing that you care about something.”
Kevlin is one of many presenters that seem to think and talk better while walking.

Mattias Göransson from Sony Ericsson presented how they have started to use Exploratory Testing with Session-Based Test Management with a Heuristics twist (the original title was “Heuristic Based testing”)
He had borrowed material from Michael Bolton (who has borrowed from James Bach), “with pride”, and explained the Heuristics concept and the flavors Guideword, Trigger, Model, Process.
“We spend more time on test execution and – SURPRISE – we find more bugs.”
They have about 25.000 requirements on a mobile phone, and uses test cases for the major areas, and Exploratory Sessions with Heuristics to get the depth, and higher coverage from the user’s perspective.
Unfortunately he wasn’t allowed to share their heuristics, but he told about the fantastic Staffanstorp Heuristic: if the phone works at a particular place in Staffanstorp, it probably works well almost everywhere.
He explained their heuristics as “experiences they have gained” and pointed out that the heuristics are reviewed after each session.
Other tips: Time-boxing of debriefings is very important.

Nikolai Tillman of Microsoft presented PEX (http://research.microsoft.com/Pex/); a tool that helps you explore single-threaded .NET code by automatically creating unit tests.
The tool analyzes the code and creates representative sets of input values; with 100% Code Branch Coverage.
I think this can be a really good help in getting started with some basic unit tests (and avoid null exceptions); and also a way of learning special cases in code you are re-using.
As merit it should also be mentioned that PEX has been used in .NET components, which resulted in bugs that were fixed.
The tool exists as Academic license for Visual Studio 2008; and early versions of Visual Studio 2010.
“dynamic symbolic execution”

The last presentation of the day before my train left was Pradeep Soundararajan (http://testertested.blogspot.com/) from India. He told his own stories, about how he got fired from a company because he didn’t meet the 95% pass rate of test cases (He found too many bugs!); about very successful testing in a team; and about changing the culture at a company that ran 3000 test cases every week, but never found any bugs.
A very nice end of the day from this member of the context-driven community.
“What is everything? It is secret.”
“Think about the customer’s customer”
“Scripts make people go mad, and sad”
“Using good practices in the right context”
“The brain is an important tool that you should consider using.”

It was a nice day at a nice conference which also included a discussion (or lecture?) in the hallway with James Bach about the schools of testing concept. I was pretty upset with him after some e-mail stuff, but in person I was pleasantly surprised. In his eyes, you can see that he is a really nice person; a main reason for his provocations is that he learns best by arguing; and probably thinks that other people does that as well.

Well, this was enough conferencing for me in a while; now it’s back to the wonderful world of parental leave.

Being creative is hard when your consciousness is turned on.
Why?
Because your consciousness is an alert cognitive state in which you are aware of yourself in your situation. It is a filter, that not only filter out mostly of the incoming information, but also tries to only deal with information that is important in the aspect of fit into the society. I.e., the social life gets prio 1.

So in order to be successfully creative, you need to train on how to step out of your conscience and enter the world of creativity.

I have realized that composing music is one of my actions to enter this world. When I am in it, I am not conscious. Only creating stuff without reflecting upon myself or awareness of how this music would be accepted by others. This might happen afterwards, but not in the middle of a session. The session is purely an creative session.

There are other actions I do that put me in the same state, e.g., crossword puzzling and playing solitaire card games.
I have even realized that in order to have a successful music session, I play solitaire card games just to get in the proper mode! I.e., I have trained myself into step out of consciousness. And since it is my decision to do so, it is a conscious way to step out of my consciousness.

The same goes for testing. The best and most creative test sessions I have had, have been those where I haven’t reflecting upon myself and my situation. All focus is on processing information and finding out new things.
Not sure if I have found any good way of entering this mode though.
Playing solitaire card games at work is not really socially accepted, yet…
Perhaps compose music might be possible to do? People love those kind of creative skills. That might be accepted…