thoughts from the test eye

The delegates of the first Swedish Workshop on Exploratory Testing were:
Torbjörn Ryber, Simon Morley, Christin Wiedemann, Petter Mattsson, Anders Claesson, Oscar Cosmo, Johan Hoberg, Rikard Edgren, Henrik Emilsson, Martin Jansson, Ann Flismark, Henrik Andersson, Michael Albrecht, Johan Jonasson, James Bach.
This write-up surely contains mistakes and important omissions, and it might be too heavy to read, but we bet you can jump in anywhere, read for a minute, and get something interesting out of it.

“15 people spending a weekend of their spare time to talk about Exploratory Testing!” (Andersson)
“This is the 1% of the 1%” (Bach)

Henrik Andersson – add Learning and Adaption to SBTM debriefing

Have experienced that testers stop doing debriefing after some time, which leads to lower Session Notes quality, or that they are skipped.
If it is a time issue, several debriefs can be made at once, or that testers debrief each other, or that all debriefs are made at once (30 minutes) at the end of the day.
The PROOF (Past, Results, Obstacles, Outlook, Feelings) mnemonic doesn’t support so much of coaching and learning, therefore Henrik advocates adding Learnings and Adaption (PROOF L.A.) that makes people discuss, learn, and use debriefs better (and stop skipping them.)

During debriefings you come up with new test ideas (Edgren)
Master the skill of note-taking
“the debrief formally accepts the session report” (Bach)
“The PROOF is (meta-)data regarding the session; LA is personal data (gathered from session)” (Emilsson)
“LA is not written down as a part of the debrief; it is only discussed orally” (Andersson)

The discussion went to many places, including status reports, and the need for training stakeholders if necessary.
“In my status reports, feelings are very important!” (Andersson)
“you almost need a handshake” (Morley)
Claesson shared a story from an East Asian company where the man-in-charge went to the most experienced troubleshooter and asked “Do you think the product is ready?”
“Troubleshooter as a career path in testing?” (Jansson)

debriefs can differ a lot depending on tester, area, type of work, experience etc.
a project manager should care about the team members’ long-term development
“testing is a learning experience in itself” (Claesson)
“don’t add things to models that already are good” (Mattsson)
“I love lists” (Wiedemann)
“do debriefs with a developer” (Flismark)
“break patterns to see what happens” (Andersson)
“are you satisfied with testers that don’t want/know how to learn?”

“it’s not a tool to fill the database” (Jansson)
As a single tester in a SCRUM team, you can have testers from different teams debrief each other.
“personal debriefing” is for the experienced
for inexperienced testers you don’t want to miss the coaching opportunity of debriefings
if you take away debriefs you might get too much freedom?
This area can be further analyzed and discussed, especially Group Debriefs.

Petter Mattsson – From 10.000 test cases to SBTM

At UIQ, producer of mobile phone software, they were running 10.000 manual test scripts, and did not find many bugs. These were instead found by customers…
Petter went to the Rapid Software Testing course and his conception of the world changed.
They decided to start using Exploratory Testing with Session-Based Test Management.
They got the opportunity to do a pilot that resulted in more, and more interesting, bugs.
A key to getting the Go decision, was a 2 hour workshop where all managers discussed questions like What is Quality? What is Testing?
They also did an exercise that showed the difference between scripted and exploratory testing.
The workshop was attended by all, the invitation was sent by a manager.

Michael Bolton did the RST course for all testers, that got inspired and motivated.
“RST with bonus day is great”
“we start with people that were hired two weeks ago, without testing experience”
They started with pair testing only, and then mixed with alone-testing.
Mixed different personality types, testers/developers, experienced/inexperienced.
“in the beginning you need to push them, but not after a while”
They developed a web-based solution for session report management (based on Perl tool)
They kept 500 test cases for verifying the basic functionality, and threw away the rest.
“we always need to do that mix” (of ET and ST)
They had really good results with the new approach, but a big phone comapny bought UIQ’s customer (that also was an owner), and the company was shut down.

Common transition problem: “ET is nice, but we have this problem, so we need to get back to the test cases.”
At another company, Petter has seen resistance to transition from testers, and also from managers, that don’t allow time to be spent.
“Just sit with them and test, let it grow organically”
“Why wait so many years for improving the way you work? Why wade in the mud when you can just step out?” (Jansson)
“They are not interested in testing; testers are used as a blame-shield” (Bach)
“do more of the robot dance” (Bach)
“If noone admits that there is a problem, noone owns the problem, and there is nothing to resolve” (Emilsson)
“ET helps you find important problems quickly”
“I’d like to have the testers off-site for a month, and just talk to them” (Mattsson)
“run the same test title, but without the steps”
heavy pain gives motivation
“They need to say: I am an alcoholic. My testing strategy really sucks” (Claesson)
“all organizations are perfectly designed to get what they get”
“small-scale change with one word: serendipity; scripted testing with deviations is a good start” (Edgren)
“I just removed the steps from the scripts, nobody noticed” (Cosmo)
Is RST the silver bullet? No, a money machine 😉
“I’ll talk to one, that talk to others, and it spreads all around” (Mattsson)
It takes time to grow an exploratory testing team.
“If you say it will be hard, it will be hard, it’s a self-fulfilling prophecy” (Claesson)
“Improvisational Exploratory Scripted Testing” (Bach)
“What’s good in these test cases live on in your heart. You shouldn’t tell people to throw away something that is useful.” (Bach)
The scripted test become a ceremonial testing.

“it is the tester’s responsibility to improve” (Many)
“we should have a class: how to evangelize testing” (Bach)
You evangelize by giving real examples, from your own organization.
“the important thing is to talk about good testing” (Morley)
“It is not necessary to create a revolution, sometimes an evolution is better” (Jansson, Emilsson)
“motivation is the most important factor, a motivated team can achieve whatever they want” (Emilsson)
if nothing helps, you should use hard facts – bug stories, from real customers
take your most painful bugs, and talk about them.
FDA recalls often say “under certain circumstances”; that’s when you need Exploratory Testing (Bach)
http://www.fda.gov/MedicalDevices/Safety/RecallsCorrectionsRemovals/ListofRecalls/default.htm

Christine Wiedemann – starting with SBTM

They are an autonomous test group of 2 persons, that works with external customers.
Customer A – many requirements, a couple of hundred test cases, 3 weeks to execute.
Customer B,C,D – no requirements.

Customer A found showstopping bugs after production, and they realized:
“what we do doesn’t work; we have to do better testing”
In March/April they stopped writing test cases.
Inspired by SBTM, they started writing test execution notes in Word, sometimes working in pair.
“Now, we’re having fun when we are testing”, and they find the defects before production.
They are cooperating with developers and customers, and have even done ET workshops with the enemy (customers)
Customers are doing ET on the delivery, they have even taken the RST course!
“Management still doesn’t care what we do.”
We have a better understanding of the product and current high-risk areas.
Went from “trying to reach coverage” to “trying to find defects before production”
shared responsibility for quality

So what is left to do:
* more structured environment
* more documentation
* better time-boxing
* automation

“unfortunately, developers are too creative, and new features appear”
“I don’t need requirements. I love it.”
They use diagrams that describe the functionality (yEd)
they report bugs to the “ether”
“I have stopped being personally attached to bugs. Stopped arguing for bugs, instead telling developers ‘I saw something strange…'” (Ryber)
you can also say “this one is probably too difficult…”
Reality Steam Roller Method – let them go into suffering, but help them (Bach)
HICCUPPS – “I invented the name. I noticed people doing this.” (Bach)
Regarding more structured environment: “You are an explorer, explore a project and try to find out what might be important during a project’s lifecycle; and use this as a checklist for deciding on what to do and when.” (Emilsson)

granularity of Session notes may vary, richer reports have more information, but are harder to read.
“Sometimes I write down the humidity in the room” (Wiedemann)
Why do you want structure? Rather, be more aware of the structure.
“be the author, not the victim”
“unserendipity – to get through the test case without finding bugs” (Edgren)
“focus on what you want to achieve” (Albrecht)
“most important traits are awareness and willingness to learn” (Claesson)
trigger yourself for new test ideas
Anders Claesson shared a story that started as a tester learning about customer’s usage had ripple effects and evolved to a very good customer relation.

Ann Flismark – SBTM and KPI??

Went to STARWEST and got inspired.
Started with SBTM in December 2009, had problems integrating it in existing system, and are working on their own tool.
“what I prepared doesn’t make sense anymore”, so instead the focus was on requests for KPI, maybe they can help us as testers as well?

“Just say no” (All)
“Do testers feel productive?” (Edgren)
“How good do you think the product is?”
“tell the manager you will make something up” (Emilsson)
“you need to learn how to do status reporting” (Bach)
The true problem: how to deal with manager requests you don’t think are meaningful.

James Bach – Experience report involving Thread-Based Test Management

At a project, it was impossible to stick to completing the sessions.
There were many and continuous interruptions, and tasks could not be completed due to various reasons, “a bunch of pots boiling”.
The simple idea for this is: Thread-Based Test Management
“work backwards from the status report you want to be able to give”
“monitors status of test activities”
Why name it? So you can say “we switched from a session-based approach to thread-based.”
“This is such a simple idea. Too simple for a book, maybe a pamphlet.”
compare activity-based, artefact-based, and metrics-focused management

Is this the same as Kanban, without limiting work in progress?
No, key idea is “you acknowledge the fact that test activities rarely are finished” (Edgren, Emilsson)
In testing, you can’t always check off items on your To-do list (which (together with throughput) is the point of Kanban)
“the simple (and hard!) thing: think of things I’m working on” (Emilsson)
get out of the “Are you done yet?” trap
Waterfall and V-model might have started as jokes?
“project management tool focused on activities that rarely are finished”
Test Storytelling Tool (test management tools only handles test cases, not stories)
Are there any other professions where you want to transform activity results to status reports?
“don’t want to over-identify this until I have talked to you” (Bach)
which are areas of importance that need to be developed?

Cosmo uses a web forum for TBTM (thetesteye have used Word)
Mind Manager could be a tool, you can use icons to filter with. Tags or table columns for other tools.
“Do you feel you are trapped by SBTM or that SBTM is your silver bullet?” (Jansson)
James said he could get biased by SBTM, but “there are no silver bullets”, Fred Brooks
Common mistake: transform the map (model) to a list that should be Done (this doesn’t fit ongoing activities) (Emilsson)
There are many situations where it doesn’t matter if you can measure if you’re done.

“Testing doesn’t deliver anything except information about the product. Information that continuously emerges. Threads describe activities that lead to information that is part of our report” (Edgren)
There are (at least) two stories; one about the testing, and one about the product.
Story-based test management – threads are weaving into a report that is the story about the product
Test framing – thread design
Testing transforms a naive story about the product to a sophisticated product story. (Bach)
naive rumors -> testing -> empirically grounded information
different threads can be executed with one test activity
“twist together into a chord”
go out of the abstract world, and visualize testing
“everybody is already doing TBTM”
developer to a tester struck by the process: “don’t worry, we’re not gonna do anything they say anyway”
“you can all say ‘James Bach says I am a world expert in thread-based test management.'” (Bach)

Everybody was very happy with the peer conference, and let’s end by quoting Andersson’s check-out: “I’m pretty damn sure we have a bright future.”

/Rikard, Henrik, Martin