The delegates of the first Swedish Workshop on Exploratory Testing were:
Torbjörn Ryber, Simon Morley, Christin Wiedemann, Petter Mattsson, Anders Claesson, Oscar Cosmo, Johan Hoberg, Rikard Edgren, Henrik Emilsson, Martin Jansson, Ann Flismark, Henrik Andersson, Michael Albrecht, Johan Jonasson, James Bach.
This write-up surely contains mistakes and important omissions, and it might be too heavy to read, but we bet you can jump in anywhere, read for a minute, and get something interesting out of it.

“15 people spending a weekend of their spare time to talk about Exploratory Testing!” (Andersson)
“This is the 1% of the 1%” (Bach)

Henrik Andersson – add Learning and Adaption to SBTM debriefing

Have experienced that testers stop doing debriefing after some time, which leads to lower Session Notes quality, or that they are skipped.
If it is a time issue, several debriefs can be made at once, or that testers debrief each other, or that all debriefs are made at once (30 minutes) at the end of the day.
The PROOF (Past, Results, Obstacles, Outlook, Feelings) mnemonic doesn’t support so much of coaching and learning, therefore Henrik advocates adding Learnings and Adaption (PROOF L.A.) that makes people discuss, learn, and use debriefs better (and stop skipping them.)

During debriefings you come up with new test ideas (Edgren)
Master the skill of note-taking
“the debrief formally accepts the session report” (Bach)
“The PROOF is (meta-)data regarding the session; LA is personal data (gathered from session)” (Emilsson)
“LA is not written down as a part of the debrief; it is only discussed orally” (Andersson)

The discussion went to many places, including status reports, and the need for training stakeholders if necessary.
“In my status reports, feelings are very important!” (Andersson)
“you almost need a handshake” (Morley)
Claesson shared a story from an East Asian company where the man-in-charge went to the most experienced troubleshooter and asked “Do you think the product is ready?”
“Troubleshooter as a career path in testing?” (Jansson)

debriefs can differ a lot depending on tester, area, type of work, experience etc.
a project manager should care about the team members’ long-term development
“testing is a learning experience in itself” (Claesson)
“don’t add things to models that already are good” (Mattsson)
“I love lists” (Wiedemann)
“do debriefs with a developer” (Flismark)
“break patterns to see what happens” (Andersson)
“are you satisfied with testers that don’t want/know how to learn?”

“it’s not a tool to fill the database” (Jansson)
As a single tester in a SCRUM team, you can have testers from different teams debrief each other.
“personal debriefing” is for the experienced
for inexperienced testers you don’t want to miss the coaching opportunity of debriefings
if you take away debriefs you might get too much freedom?
This area can be further analyzed and discussed, especially Group Debriefs.

Petter Mattsson – From 10.000 test cases to SBTM

At UIQ, producer of mobile phone software, they were running 10.000 manual test scripts, and did not find many bugs. These were instead found by customers…
Petter went to the Rapid Software Testing course and his conception of the world changed.
They decided to start using Exploratory Testing with Session-Based Test Management.
They got the opportunity to do a pilot that resulted in more, and more interesting, bugs.
A key to getting the Go decision, was a 2 hour workshop where all managers discussed questions like What is Quality? What is Testing?
They also did an exercise that showed the difference between scripted and exploratory testing.
The workshop was attended by all, the invitation was sent by a manager.

Michael Bolton did the RST course for all testers, that got inspired and motivated.
“RST with bonus day is great”
“we start with people that were hired two weeks ago, without testing experience”
They started with pair testing only, and then mixed with alone-testing.
Mixed different personality types, testers/developers, experienced/inexperienced.
“in the beginning you need to push them, but not after a while”
They developed a web-based solution for session report management (based on Perl tool)
They kept 500 test cases for verifying the basic functionality, and threw away the rest.
“we always need to do that mix” (of ET and ST)
They had really good results with the new approach, but a big phone comapny bought UIQ’s customer (that also was an owner), and the company was shut down.

Common transition problem: “ET is nice, but we have this problem, so we need to get back to the test cases.”
At another company, Petter has seen resistance to transition from testers, and also from managers, that don’t allow time to be spent.
“Just sit with them and test, let it grow organically”
“Why wait so many years for improving the way you work? Why wade in the mud when you can just step out?” (Jansson)
“They are not interested in testing; testers are used as a blame-shield” (Bach)
“do more of the robot dance” (Bach)
“If noone admits that there is a problem, noone owns the problem, and there is nothing to resolve” (Emilsson)
“ET helps you find important problems quickly”
“I’d like to have the testers off-site for a month, and just talk to them” (Mattsson)
“run the same test title, but without the steps”
heavy pain gives motivation
“They need to say: I am an alcoholic. My testing strategy really sucks” (Claesson)
“all organizations are perfectly designed to get what they get”
“small-scale change with one word: serendipity; scripted testing with deviations is a good start” (Edgren)
“I just removed the steps from the scripts, nobody noticed” (Cosmo)
Is RST the silver bullet? No, a money machine
“I’ll talk to one, that talk to others, and it spreads all around” (Mattsson)
It takes time to grow an exploratory testing team.
“If you say it will be hard, it will be hard, it’s a self-fulfilling prophecy” (Claesson)
“Improvisational Exploratory Scripted Testing” (Bach)
“What’s good in these test cases live on in your heart. You shouldn’t tell people to throw away something that is useful.” (Bach)
The scripted test become a ceremonial testing.

“it is the tester’s responsibility to improve” (Many)
“we should have a class: how to evangelize testing” (Bach)
You evangelize by giving real examples, from your own organization.
“the important thing is to talk about good testing” (Morley)
“It is not necessary to create a revolution, sometimes an evolution is better” (Jansson, Emilsson)
“motivation is the most important factor, a motivated team can achieve whatever they want” (Emilsson)
if nothing helps, you should use hard facts – bug stories, from real customers
take your most painful bugs, and talk about them.
FDA recalls often say “under certain circumstances”; that’s when you need Exploratory Testing (Bach)
http://www.fda.gov/MedicalDevices/Safety/RecallsCorrectionsRemovals/ListofRecalls/default.htm

Christine Wiedemann – starting with SBTM

They are an autonomous test group of 2 persons, that works with external customers.
Customer A – many requirements, a couple of hundred test cases, 3 weeks to execute.
Customer B,C,D – no requirements.

Customer A found showstopping bugs after production, and they realized:
“what we do doesn’t work; we have to do better testing”
In March/April they stopped writing test cases.
Inspired by SBTM, they started writing test execution notes in Word, sometimes working in pair.
“Now, we’re having fun when we are testing”, and they find the defects before production.
They are cooperating with developers and customers, and have even done ET workshops with the enemy (customers)
Customers are doing ET on the delivery, they have even taken the RST course!
“Management still doesn’t care what we do.”
We have a better understanding of the product and current high-risk areas.
Went from “trying to reach coverage” to “trying to find defects before production”
shared responsibility for quality

So what is left to do:
* more structured environment
* more documentation
* better time-boxing
* automation

“unfortunately, developers are too creative, and new features appear”
“I don’t need requirements. I love it.”
They use diagrams that describe the functionality (yEd)
they report bugs to the “ether”
“I have stopped being personally attached to bugs. Stopped arguing for bugs, instead telling developers ‘I saw something strange…’” (Ryber)
you can also say “this one is probably too difficult…”
Reality Steam Roller Method – let them go into suffering, but help them (Bach)
HICCUPPS – “I invented the name. I noticed people doing this.” (Bach)
Regarding more structured environment: “You are an explorer, explore a project and try to find out what might be important during a project’s lifecycle; and use this as a checklist for deciding on what to do and when.” (Emilsson)

granularity of Session notes may vary, richer reports have more information, but are harder to read.
“Sometimes I write down the humidity in the room” (Wiedemann)
Why do you want structure? Rather, be more aware of the structure.
“be the author, not the victim”
“unserendipity – to get through the test case without finding bugs” (Edgren)
“focus on what you want to achieve” (Albrecht)
“most important traits are awareness and willingness to learn” (Claesson)
trigger yourself for new test ideas
Anders Claesson shared a story that started as a tester learning about customer’s usage had ripple effects and evolved to a very good customer relation.

Ann Flismark – SBTM and KPI??

Went to STARWEST and got inspired.
Started with SBTM in December 2009, had problems integrating it in existing system, and are working on their own tool.
“what I prepared doesn’t make sense anymore”, so instead the focus was on requests for KPI, maybe they can help us as testers as well?

“Just say no” (All)
“Do testers feel productive?” (Edgren)
“How good do you think the product is?”
“tell the manager you will make something up” (Emilsson)
“you need to learn how to do status reporting” (Bach)
The true problem: how to deal with manager requests you don’t think are meaningful.

James Bach – Experience report involving Thread-Based Test Management

At a project, it was impossible to stick to completing the sessions.
There were many and continuous interruptions, and tasks could not be completed due to various reasons, “a bunch of pots boiling”.
The simple idea for this is: Thread-Based Test Management
“work backwards from the status report you want to be able to give”
“monitors status of test activities”
Why name it? So you can say “we switched from a session-based approach to thread-based.”
“This is such a simple idea. Too simple for a book, maybe a pamphlet.”
compare activity-based, artefact-based, and metrics-focused management

Is this the same as Kanban, without limiting work in progress?
No, key idea is “you acknowledge the fact that test activities rarely are finished” (Edgren, Emilsson)
In testing, you can’t always check off items on your To-do list (which (together with throughput) is the point of Kanban)
“the simple (and hard!) thing: think of things I’m working on” (Emilsson)
get out of the “Are you done yet?” trap
Waterfall and V-model might have started as jokes?
“project management tool focused on activities that rarely are finished”
Test Storytelling Tool (test management tools only handles test cases, not stories)
Are there any other professions where you want to transform activity results to status reports?
“don’t want to over-identify this until I have talked to you” (Bach)
which are areas of importance that need to be developed?

Cosmo uses a web forum for TBTM (thetesteye have used Word)
Mind Manager could be a tool, you can use icons to filter with. Tags or table columns for other tools.
“Do you feel you are trapped by SBTM or that SBTM is your silver bullet?” (Jansson)
James said he could get biased by SBTM, but “there are no silver bullets”, Fred Brooks
Common mistake: transform the map (model) to a list that should be Done (this doesn’t fit ongoing activities) (Emilsson)
There are many situations where it doesn’t matter if you can measure if you’re done.

“Testing doesn’t deliver anything except information about the product. Information that continuously emerges. Threads describe activities that lead to information that is part of our report” (Edgren)
There are (at least) two stories; one about the testing, and one about the product.
Story-based test management – threads are weaving into a report that is the story about the product
Test framing – thread design
Testing transforms a naive story about the product to a sophisticated product story. (Bach)
naive rumors -> testing -> empirically grounded information
different threads can be executed with one test activity
“twist together into a chord”
go out of the abstract world, and visualize testing
“everybody is already doing TBTM”
developer to a tester struck by the process: “don’t worry, we’re not gonna do anything they say anyway”
“you can all say ‘James Bach says I am a world expert in thread-based test management.’” (Bach)

Everybody was very happy with the peer conference, and let’s end by quoting Andersson’s check-out: “I’m pretty damn sure we have a bright future.”

/Rikard, Henrik, Martin

Well, to many people this is nothing new. But still, there are a lot of testers, and indeed test leads, that still think that Exploratory Testing is a technique that can be used in testing. To some extent, it has to do with that both Cem Kaner and James Bach have used this term amongst other techniques (e.g., in the BBST course material). But they have changed and updated presentations as much as possible over the last period of time.

According to Cem Kaner nowadays, the definition of exploratory testing is “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

And this is important. You can come a long way on reaching the style of Exploratory Testing just by treating testers as intelligent people; which is one of the most important factors in the definition above. In contrast to Exploratory Testing you have Scripted Testing that, in my opinion, treats testers as dumb people or even dumb machines. I think that this approach is devastating for our profession (even though I can somehow see the need for Scripted Testing in some places).

A technique is a recipe for solving a problem, whereas a style (or approach) is a way of thinking around a theme that stretches far beyond solving a particular problem.
So when we talk about selling in Exploratory Testing to managers or project stakeholders it is not a technique that we are selling; it is rather an acceptance of a mindset where testers are treated as professional and intellectual human beings that are able to perform Sapient Testing, and particularly in an Exploratory way. It is not about stakeholders investing in a technique, it is about them showing that they have as much trust in testers as they have in other intelligent co-workers of the project.

Session based testing is most often discussed in combination with exploratory testing. The idea is to make exploratory testing more structured by using it, as James Bach expresses it. I like the whole concept about testing in sessions, thus breaking the day into chunks of work. Considering that you know that there is an infinite set of tests possible, that it is easy to get side-tracked by meetings and other activities, you can use these sessions as a way to gain focus on testing and notify others that this time is holy. You can take the whole package with what this means, but you could also start with working in sessions. Further down the road it is possible to take the transition into using the whole spectra of Session Based Testing Management (SBTM).

Based on my own observations… it seems like it is easy for observers to understand the benefits of using SBTM. The concept of focus, defocus and debrief are quite self-explaining. The initial response is positive from personel utterly new to testing. Some comments I’ve heard was that it feels like you get things done and that you can easier move on to the next task at hand. I also notice that the test group seem to collaborate more, enjoy themselves in the testing tasks and gain confidence as testers. The hard part is the continuous documentation, known as a charter, created in order to make each session accountable. I see this as a natural step in becoming a good tester. If you cannot explain what you have been doing, cannot show or present some kind of results or observations, I would question what you really had been doing.

If you are in an organisation (probably a big one) where process changes takes time and might need a long line of approval, this might be something worth trying out. It is possible to implement directly into the test team. What you put in these sessions is up to you. The mission for each test session can be many different things, such as a group of test cases or a few test ideas or risks worth exploring. Using SBTM does not have to replace the way you work today, but complement it by adding a new and, as I see it, improved way of handling the exploratory part of testing.

I’ve recently finished James Bach’s book Secrets of a Buccaneer Scholar. I liked it, but I don’t agree with all of it. As a tester, I feel that it inspires me and gives me new ideas in my way of thinking and how I perceive learning, especially self-education. I fully agree with James on that you should follow your passion. If you are able to assist those around you to that end, it will make you grow even more. In march James will be in Sweden and hold a series of courses, one of them is Self-Education for testers. It think it would be very fun and educational to attend, I will see if I can make time.

One thing that I consider after reading the book is how I can inspire my daughter to learn, test new things and to follow her passion. When she receives a new toy, I want her to explore how it works. For instance, she got a little toy puppy that execute somersaults. It had a lot of mechanic inside it and sounded like it was very fragile, if you actually played with it. The purpose of the toy was probably just to watch it. I dislike such toys. I asked my daughter how she thought it worked. She did a somewhat exploratory, destructive test by enabling the puppy to do the somersault, then directly afterwards hugged it tightly. There was a popping sound in the mechanic as the puppy tried to execute the somersault, while being held secure. After that test it was not possible for the puppy to somersault, rather it performed a half one and landed on its nose. I applauded my daughters destructive sense of testing of the toy. Translating that into testing terms, she prioritized which test to start with and considered what was the biggest chance a user would do then executed that. One test to render the system useless. Wonderful!

I think we have a lot to learn (or perhaps relearn) from our children.

I use Jerry Weinberg’s definition of quality: “Quality is value to some person”; and I use Cem Kaner’s extension to the definition so that it becomes “Quality is value to some person (that matters)”…

I.e. quality is inherently subjective. And there are a lot of persons that are affected by software that we produce… With this in mind it becomes hard for a tester to stay focused when there are so many persons with opinions that could matter; but if we can find out “who matters” we decrease the number of possible values to care about. Still, this will leave us with several important values that need to be taken into account when testing the product.

So how can we testers deal with that?

You could do a role play when testing and put on someone’s hat during the test session; or you could let real users test the product and let them have a say about what they find.
But for a skilled tester it is more about being multidimensionally subjective and think as several persons at the same time.

This means that a lot of values, beliefs and preferences are taken into account which might matter. Not as an average, but as several independent quality dimensions that has (more or less) importance. The hard thing is to know when a value is threatened and for which (type of) person that is affected; and if this matters at all.
I.e., it is a matter of questioning “is there a problem here?” constantly and try to pair a potentially threatened value with its corresponding person. And if this problem threatens a value for some person that matters, we have found a bug. This corresponds to the definition of bug from Cem Kaner “A bug is something that threatens the value of a product”.

Much of this happens automatically for many of you skilled testers out there; when I thought of it recently I realized that this is something I do more and more and hopefully I am improving this skill each day. This is a great skill to have when testing software!

Anyone having any thoughts on this?
Have you experienced this yourself?
If not, does it sound like an interesting thing to examine? Would this be helpful to you?

Cheers,
Henrik

Update 2009-09-14: According to comment from Michael Bolton, see below, the quotes that I said belonged to Cem Kaner are both quotes from James Bach. I apologize for referencing wrong person.

I spent one day at Øredev 2008 (http://www.oredev.org) since they invited me to give the Where Testing Creativity Grows (http://www.thetesteye.com/papers/where_testing_creativity_grows.doc) presentation.
I arrived ten minutes after the start of James Bach’s keynote The Renaissance Thinker, where he argued that 1972 (Chapel Hill) ruined good software testing.
People started focusing too much on templates, processes, best practices; people forgot about people.
But things are getting better since the Exploratory Testing has been coined and used with more respect.
We also have Agile that is using cyclic learning, in opposite to Waterfall and V-Model where no learning is modeled.
He likes it if people say “James, maybe you are one of the mystics?” and explained his self-learning style of Buccaneer-Scholar.
“Software Engineering is primarily social.”
He has a lot of good things to say; and he presents them in a capturing manner.

This conference is nice since it includes developers, testers, project mangers et.al.
But I stayed at the Test Track all day; starting with Isabel Evans, who is interested in processes and people; and stresses that it is the people in the team, that needs to define the process.
Quality is difficult, since it has so many attributes; the customers view being the most important.
Isabel is criticizing the “standards” from the inside, but think it is important to measure things and report back, e.g. by showing how much money is saved by doing software testing.
“Agile works since it is cross-functional; we work together.”
She also broadened the spectra of our risk thinking; which too often focus on internal risks. But there are Contractual or Regulatory Risks as well as Social Ethical Risks. It is good to think bigger.

Then it was time for my own presentation with about 60 attendees. I had an interesting start with technical problems as my machine wouldn’t use the projector after I launched a movie inside PowerPoint (Yes, I use Vista.)
So I had to improvise a bit and focus on the audience instead of my slides.
After this the presentation went well; and I was happy that an attendant could tell about their paired testing (both scripted and exploratory) where they found three times as many bugs (plus the learning!)
A good question after the presentation was “When do you know you have used enough creativity?” with the answer “Well, you don’t…”

After a lunch break Kevlin Henney talked about “Know Your Units”, with the clarification that a unit test that interacts with something (file system, database, network) isn’t a real unit test.
Programmers are responsible for unit testing (“The programmer who wrote the code is in the best position to test it”) and ”QA” for system testing. (He didn’t mention any collaboration, which I think might be the best.)
POUTing -Plain Ol´ Unit Testing – is passive.
TDD – Test-Driven Development – is active.
DDT – Defect-Driven Testing – is reactive.
And sometimes we can settle with GUTs – Good Unit Tests.
He recommends a black-box perspective for unit tests, where you focus on behaviors, or even requirements. White-box testing can end up testing that the code does what it does.
“We say that system testing takes one month, but in reality, the testing takes a few days, the rest is spent fixing bugs.”
“Testing is a way of showing that you care about something.”
Kevlin is one of many presenters that seem to think and talk better while walking.

Mattias Göransson from Sony Ericsson presented how they have started to use Exploratory Testing with Session-Based Test Management with a Heuristics twist (the original title was “Heuristic Based testing”)
He had borrowed material from Michael Bolton (who has borrowed from James Bach), “with pride”, and explained the Heuristics concept and the flavors Guideword, Trigger, Model, Process.
“We spend more time on test execution and – SURPRISE – we find more bugs.”
They have about 25.000 requirements on a mobile phone, and uses test cases for the major areas, and Exploratory Sessions with Heuristics to get the depth, and higher coverage from the user’s perspective.
Unfortunately he wasn’t allowed to share their heuristics, but he told about the fantastic Staffanstorp Heuristic: if the phone works at a particular place in Staffanstorp, it probably works well almost everywhere.
He explained their heuristics as “experiences they have gained” and pointed out that the heuristics are reviewed after each session.
Other tips: Time-boxing of debriefings is very important.

Nikolai Tillman of Microsoft presented PEX (http://research.microsoft.com/Pex/); a tool that helps you explore single-threaded .NET code by automatically creating unit tests.
The tool analyzes the code and creates representative sets of input values; with 100% Code Branch Coverage.
I think this can be a really good help in getting started with some basic unit tests (and avoid null exceptions); and also a way of learning special cases in code you are re-using.
As merit it should also be mentioned that PEX has been used in .NET components, which resulted in bugs that were fixed.
The tool exists as Academic license for Visual Studio 2008; and early versions of Visual Studio 2010.
“dynamic symbolic execution”

The last presentation of the day before my train left was Pradeep Soundararajan (http://testertested.blogspot.com/) from India. He told his own stories, about how he got fired from a company because he didn’t meet the 95% pass rate of test cases (He found too many bugs!); about very successful testing in a team; and about changing the culture at a company that ran 3000 test cases every week, but never found any bugs.
A very nice end of the day from this member of the context-driven community.
“What is everything? It is secret.”
“Think about the customer’s customer”
“Scripts make people go mad, and sad”
“Using good practices in the right context”
“The brain is an important tool that you should consider using.”

It was a nice day at a nice conference which also included a discussion (or lecture?) in the hallway with James Bach about the schools of testing concept. I was pretty upset with him after some e-mail stuff, but in person I was pleasantly surprised. In his eyes, you can see that he is a really nice person; a main reason for his provocations is that he learns best by arguing; and probably thinks that other people does that as well.

Well, this was enough conferencing for me in a while; now it’s back to the wonderful world of parental leave.