I have been using the Scripted – Exploratory Testing Continuum (For one source of this, see page 56 in http://www.ryber.se/wp-content/EssentialTestDesign.pdf ) in classes to explain how scripted testing and exploratory testing intertwines; and to explain that most of our testing is somewhere in the middle of the continuum.

But I have also had some issues with this description.

If this continuum exist, when can you say that you are doing more exploratory than scripted (and vice versa)?
How do you recognize when your testing is more exploratory than scripted (and vice versa)?

I do not believe that these questions can be answered because I think it has to do with the mindset of the person who sets the approach; and how far this particular mindset can be stretched.

The continuum is a simplified (yet useful) description of how your testing approach is a mix of the two approaches. And the simplification lies in that it can be seen as if our testing approach can be found in some point on the continuum. But I think that someone who has an exploratory testing approach is doing more or less exploratory testing; and someone who has a scripted testing approach is doing more or less scripted testing. Maybe this is how many of you already think of this?
However, I would like to visualize the differences in a slightly different way in order for me to understand this better.

Testing can be viewed from different point of views; and if I as a tester look at my testing approach I can have good use of the continuum. But the decision on which approach we should focus on is often driven by the management of the project or test team. So, from a managerial point of view, the testing approach is driven from a core dimension (interest, factor) which is more or less strict. I.e., I think that each approach can be seen as a set of continuums where each continuum consist of a certain dimension; and you will only modify these individual dimensions, not the entire approach.

Scripted Testing Continuums (some examples)

Exploratory Testing Continuums (some examples)

Let’s say that my main interest is Control, and I believe that a scripted testing approach is the best option in order to fulfill my interest. From this, if I now gradually become less stricter when it comes to my control demand, I won’t transit into an exploratory testing approach. My approach would instead be gradually less scripted.

In the same fashion, if my main interest is trust. My approach would gradually become less exploratory, but never transit into a scripted approach.

I think that this has to do with that the approach is a mindset; or rather an ideology. And it is hard for many people to change their mindset/ideology because your approach is derived from your school of thought.
If you have a scripted approach to testing, it is because of something you strongly believe in. If you have an exploratory approach to testing, it is because of something else you strongly believe in.

I spent this weekend in nice, dark, foggy Ringsjöstrand for the third Swedish Workshop on Exploratory Testing. Johan Jonasson, Ola Hyltén, Anders Claesson, Oscar Cosmo, Petter Mattsson, Rikard Edgren, Henrik Andersson, Robert Bergqvist, Maria Kedemo, Sigge Birgisson, Simon Morley.

The format is LAWST-style, which means a presentation is followed by a facilitated discussion, that goes on as long as it brings value.
The theme for this event was Teaching Testing, and the abstracts can be downloaded here.

I did the first presentation using my current assignment as teacher together with Henrik Emilsson on a 2-year higher vocational study for testers.
We have a lot of exercises to make the knowledge stick, and we primarily use open source applications to make it more real.
I shared how we find suitable applications and tie them to something we are teaching, and I showed some examples of applications we have used.
The discussion that followed lasted a couple of hours and spanned many areas: course literature, complexity of software, unreal situations, authorities, ISTQB, tools, how to teach so they learn etc.
I learn a lot by teaching, and also by discussing the teaching.

The second experience report came from Johan Jonasson, who is involved as instructor in BBST online courses.
The videos are available for free, but the best thing with the education is the realistic problem-based assignments and the peer reviews.
Discussions followed for a couple of hours and included cultural and online difficulties/opportunities, how to give feedback in writing, and a summary of the slides material:
“a condensed library of test information that will broaden your views.”

After a break, we had six eight-minute Lightning Talks, where you can’t go deep, but a lot of ideas are brought up:
Anders Claesson shared his model for making sure students really learn what is thought.
Sigge Birgisson talked about his efforts to create a quality vision/model/goal for deliverables.
Rikard Edgren wondered why no one (except Oscar Cosmo) test charisma when we all know it is important. (Emilsson came up with the name Charisma.)
Henrik Andersson showed that objectivity is an illusion, so we should focus on the inherent subjectivity in test selection, execution, interpretation.
Robert Bergqvist wondered if Exploratory Testing has been more accepted, or if it just is a word in fashion.
Simon Morley has a great idea on a Groopman-inspired book “How Testers Think”, although he denies he will write it.
Then we had dinner, bubble pool, a lot of conversations on many subjects (“lean is waste for testing”), perhaps a small beer, and a good night’s sleep.

The second day only had room for one topic, which was Simon Morley on “Mindset Changes: Changing the direction of the oil tanker”, about how to teach the “right” view on testing to both managers and testers.
Many suffer from test case counting, and they have to be cured in different ways.
Simon says to testers he don’t want to hear any numbers, and thereby force them to talk about what they have seen, and what they haven’t done.
He does presentations to managers about problems with number focus, and after that he can use dashboard-like reporting.
The most important dimension might be trust.
Consensus in the room was that gut feelings are better for making decisions, but that we don’t have the proper words to talk about this.
A lot of examples and ideas were shared, but we didn’t reach any solutions before the time was up…
“if communication isn’t good, it doesn’t matter what you do”

Eleven happy check-outs with a richer network and knowledge; SWET has charisma.

During my first paternity leave I learned sourdough baking. During the second I couldn’t help writing an ambitious paper, or a small book, about people-oriented test design, about things beyond test design techniques, close to the exploratory testing tradition.

It can be downloaded here.

It contains collections of knowledge, and generalizations of my ten years of testing the same product suite. I think it can be useful for ambitious testers that want to find any problems that might be important.

It probably is too much, theoretical, irrelevant or condense for many of you, but if you want to give it a shot I recommend the following:

* Download The Little Black Book on Test Design
* Print as double-sided A5 Booklet
* Find a quiet, comfortable place
* Read and relate to your test reality

Comments are welcome, especially additions to the collection of one hundred and three test design heuristics.

The delegates of the second Swedish Workshop on Exploratory Testing (Test Planning and Status Reporting for Exploratory Testing) were:
Henrik Andersson, Azin Bergman, Robert Bergqvist, Sigge Birgisson, Rikard Edgren, Henrik Emilsson, Ola Hyltén, Martin Jansson, Johan Jonasson, Saam Koroorian, Simon Morley, Torbjörn Ryber, Fredrik Scheja, Christin Wiedemann, Steve Öberg

Discussions on peer conferences can’t be summarized, but you can read the abstracts, Ryber’s notes, and here comes some quote highlights:

- How many test cases do you want? 800? Then let’s say we have 800. (Jonasson)
- Think “there are problems”, not “there might be problems” (Bergqvist)
- To motivate: lead by example, avoid de-motivating (Hyltén, Andersson)
- They want to see bug curves, but don’t know it going up or down is good or bad. (Jonasson)
- Rather ask “What are you afraid of?” (Jansson)
- Qualitative information is sensitive to filtering (Emilsson)
- Dialogue is important! (Morley)
- Exploratory test planning, adapt to reality. That’s how we all work. (Scheja)
- A test expert that doesn’t know anything (favorite slip of the tongue from Jansson)
- Has this story confirmed that anyone can test? (Bergman)
- Incorrect expectations: if we test, there won’t be any production problem (Ryber)
- Categorizations are good for learning, but throw them away to meet reality (Edgren)
- I always use low-tech testing dashboards, except in this case (Ryber)
- A tester is also test leader, test designer, test planner (Scheja)
- It is small raisins in a very large cake (Jonasson)
- I want to test until the world is on fire (Wiedemann)
- Care about what is most important (Emilsson)
- I invent many wheels every day (Scheja)
- I want more emphasis on which information we have and don’t have, and the confidence of the information (Koroorian)
- Beware of using the word “coverage”, rather say security, or risk (Birgisson)
- To me, coverage is a start of thinking about what to test (Bergqvist)
- If this isn’t reported “upwards”, you avoid the biggest risks (Andersson)
- The map and the creation of it is more important than the numbers on it (Öberg)
- It might look easy with a number, but it isn’t (Edgren)
- This weekend is evidence that standards don’t work. We can discuss forever, and that’s what is fun! (Emilsson)

These Lightning Talks (5 minutes including questions) were held:
* Martin Jansson on the tester’s greatest nemesis, the view that anyone, thus any idiot, can test.
* Henrik Emilsson on unjustified tests that open up to serendipity and new ideas.
* Robert Bergqvist had an experience report that showed that exploratory testing also needs planning.
* Azin Bergman on the common view that testers have lower status than other roles.
* Simon Morley on root cause analysis heuristic FICL: Framing, Information Gathering, Consensus, Learning.
* Rikard Edgren on Binary Disease – our tools have shaped way too computeresque theories.
* Christin Wiedemann on open-ended plans without fixed scope.
* Sigge Birgisson shared an experience report with developer collaboration under the radar.
* Steve Öberg led a brainstorm on testing analogies with story from the Aeneid about not following the Delphi Oracle.
* Ola Hyltén showed the Johari window that can enable better communication between leaders and team members.
* Torbjörn Ryber led a discussion about doing a context-driven conference in Sweden.

It is very easy to organize a workshop like this when you have 15 motivated, passionate testers.
You just need a venue with few distractions, a theme to focus, an agenda and discussion rules, plus food and drinks.

Looking forward to SWET3!

Once again I have spent the weekend with members of the cream of Swedish testers. This time The Test Eye trio consisting of Henrik Emilsson, Martin Jansson and Rikard Edgren were the hosts.

The theme was Exploratory Testing and Planning and we managed to keep the discussions within that scope most of the scheduled sessions. The informal test talk, music and drinking session from 18.30 to 03.30 encompassed a rich variety of discussions mostly outside that theme. Robert brought a fantastic instrument called the beat box (I think?), Rikard the guitar and the rest of us had guitar picks, maracas and music talent to some degree. The starting act was “The  Spice Song” composed and performed by multi talented philosopher-baker Rikard. Other highlights were “My Sharona” performed by many of us, later followed by interpretations of Cornelis Vreswijk and early in the morning sometime a try at “Bark at the moon”. The barking was considerable but I prefer the original version with Ozzys falsetto. And let us not forget the African freedom and tribal songs performed by the “missionary men”. Music is so much fun! If we ever start a band it will be called the Testicles.

Johan

But let us start with the conference pass. First session was Johan Jonasson from House of Test. He told us of the success he had with two young girls from help desk. I bet that sentence caught your attention! Well it was just as exciting as it sounds – it is all about testing. Jonas’ task was to manage the testing of a consumer product application and the resources he was assigned was two support people. He pointed out that they lacked education in IT and testing but were very motivated. He had too little time in order to prepare any detailed instructions. Test instructions consisted of some form of user stories created by himself and scenarios built by these. They tested together following the instructions and were coached by himself at a couple of meeting each day. They turned out to be great at taking notes and very good at finding problems. In the following open season we concluded that they did not lack education at all. They were used to taking support calls and analysing the caller’s problems so they had both product knowledge and experience on analysing and describing problems in text. Some reflections was that motivation is a very important factor and that less detail control of curios and motivated testers made them perform better. I can think of at least one client that uses support personnel but detailed script that should try the approach with a more exploratory format of working. As first sessions often do, it lasted for almost four hours and the discussions were never once uninteresting.

Tobbe

Number two on the list was myself with an experience report of testing in a loosely controlled and volatile project. Given the role as tester on the project I gradually moved into project management, requirement elicitation at the expense of much less testing than planned. Some of the things I did was to create a product backlog, organise weekly scrum meetings, create an effect map and a status graph. My overarching goal was that we actually deliver something to the customer and whatever needs there were somebody needed to take care of them. Success factors was that I was not only allowed but encouraged to assume new responsibilities by the other project members and that I find it interesting to take on other tasks than testing. The downside was that I had less time testing and, to be honest, less motivation to test. Maybe I would have tested better if I was given full-time on the project but when that was not the case I prioritised other tasks in order to move forward. It should be the testers goal to do whatever is needed to bring the project forward. I see similarities with the Scrum thoughts that we have less specialised roles. We had discussion on whether the artifacts I created was testing or not but the question is if it matters as long as they are important and they get done?

Fredrik

Fredrik Scheja of Sogeti was the next presenter. He told us about his success with an exploratory testing approach on a large system with frequent releases. Every tester assumes responsibility for analysing, test planning and execution of one or more items at a time. One of the dominating discussion in open season was the fact that he claimed that he built this approach on TMAP. Since most of the participants claim to be members of the context-driven school this is quite a daring statement to make at a peer conference. TMAP is clearly factory school which is the opposite of the context-driven approach. While I think none of us doubt the success of the work process used, many of us claim that it is not really TMAP-based just because you pick some parts that you find usable and then tweek them to fit what you really want to do. I think it was Johan that used the metaphor “Picking very few raisins out of a very large cake”. James Bach tweeted that “If you take a nice cake and drop it in a mud puddle, don´t bother with the raisins”.  Henke Andersson attacked the claim that the 12 step checklist for test planning really was the best to use for all planning in all situations and wanted to see a local adaption for the project that was very unTMAP(is there such a word, well now there is!). The discussion continued for another hour on Sunday morning for all of us except Ola Hyltén that by mistake(?) forgot to set his alarm when he went to bed already at 3.30. We thank you Ola for giving us an opportunity to make fun of you J

Lightning Talks and Conference plans

Last session Saturday was a number of lightning talks that were entertaining but to be honest I don´t remember much of the ten five minute talks. If someone else has a record or want to say something about them – be my guest. I do remember the last one where Henke told me to remind the group of our idea to arrange a context-driven conference in Sweden within the next year. All said it was a great idea and many wanted to help out. Our first suggestions was that we need a year to plan and arrange, it should take place in a larger city – probably Stockholm. Size could be 300 participants. Goal is not to make a fortune but not loosing money. Planning for a small profit gives us some room for unexpected costs. We would like to really focus on the context-driven community, only context-driven talks and not being sponsored by any certification organisation. We want to have some key players from the USA and certainly from the rest of the world as well. We may want to consider some tracks only in English and some in Swedish. All thoughts welcome.

Dinner

Dinner was a seafood buffet at the toll house by the sea. Since three of our four vegetarians have redefined fish and shellfish as vegetables they happily dug into the buckets of crabs, shrimp and langoustes (or whatever the English name for havskräfta is).

Martin said he was disappointed there was not a pool and agreed that he could possibly assume some responsibility for that fact since he was the one that booked. We focused on the music part instead as told before.

Saam

Sunday morning started out with another hour of open season on Fredrik followed by Saam that explained his goals to change testing and reporting on the large company where he works. We spent a couple of hours discussing green/yellow/red versus 1-5 versus happy faces and sad faces. The goal was to move from numbers – which say very little – to a more qualitative approach. This can be quite a challenge in an international and global organisation. We all look forward to learn about the results of Saams intentions in the future.

The happy ending

After some hugging, handshaking and some tears we left for the mainland. Yeah, cause I forgot to tell you we stayed at Chicken Island outside Gothenburg.

Henke Andersson has promised to arrange SWET3 in Malmö this fall. He mentioned that it will focus on ET standards and the need for ET certification…or maybe not. One subject that I would like to discuss more is teaching testing.

Additional info on Twitter #SWET2

The delegates were: Christin Wiedemann, Torbjörn Ryber, Azin Bergman, Fredrik Scheja, Henrik Andersson, Johan Jonasson, Ola Hyltén, Sigge Birgisson, Simon Morley, Rikard Edgren, Henrik Emilsson, Martin Jansson, Steve Öberg, Robert Bergqvist, Saam Koroorian.

Monday

Henrik started his journey by car from Karlstad and went down to Gothenburg to join with Martin. Both of us were going to take the train down to Copenhagen. Not surprisingly there were delays and the train was cancelled… Instead we headed back to Martin’s house and loaded Henrik’s car and took off. After several hours drive we arrived to Copenhagen, where we went directly to Bella Center and the TestLab at EuroSTAR.

This was the first time we met James Lyndsay and Bart Knaack in person; we had only talked with them over Skype/phone. So we all started with saying hello but there were really no time to sit down and have a cup of coffee – we needed to get to work straight away.

One of the burning issues to resolve the week before the conference was to bring client machines. Henrik was able to bring four laptops (thank you Compare Testlab and Know IT for sponsoring with this!) and Martin one. We worked in the TestLab for a few hours setting up and testing the two networks (James had one and Bart had another) with the servers connected to them.

Each network consisted of a bug tracker (Mantis), a server with OpenEMR, a server with a WebShop based on OsCommerce as well as a link to download a version of FreeMind.

When the conference center was getting near closing hours we rounded up and identified tasks needed to be done for the next coming days. We were assigned to setup and fill both the OpenEMR and the WebShop with new test data. We determined that we first should create a data model in FreeMind; both for reviewing before inserting the data and also so that we could print out the model and use it as a means to discuss with participants in the TestLab. With the model at hand, we thought the participants would also be able to identify test ideas of their own that they could try out.

We packed up everything and took Henriks car together with James and Bart. Half an hour later we were down in the hotel bar for dinner. We first met up with Steve Öberg, Rikard Edgren, Björn Karlsson, and Stefan Thoresson; and later on Markus Gärtner, Neil Thompson & Michael Bolton and some more joined our table and we were discussing testing long into the night.

Tuesday

At 8.45 we met up with James and Bart at the hotel and took a cab to the conference center where we setup the TestLab again. We started on the data models while James and Bart worked on setting up the details of the two networks. Meanwhile the sponsors were installing their applications on the client machines. Late into the afternoon we were starting to get ready with the new data for both systems. During the afternoon several people were visiting us and some of them started testing the applications. In fact, the first bug entered in the bug system was the one that won the prize “Best bug” – Markus Gärtner found it pretty fast…
Around 18:00 we were able to move into the room where the TestLab should be. We moved in all servers, routers and clients and setup the power for it all. Funny thing is that we didn’t find more than one electricity socket, so all machines were hooked up on this only one and we decided to load test the electricity over night.
During the evening we attended the Rebel Alliance (or Danish Alliance) for 11 interesting lightning talks. The discussions went on after that long into the night. But to our defense we must say that we got in bed in time.

Wednesday

Despite that we got in bed fairly early, we both overslept and missed the grand opening of the TestLab… Such a good start!
But no time to cry over that; we dug in and assisted in welcoming and introducing people that entered the TestLab and told them about it and how to use it; as well as guiding them to start testing. Bugs were reported and the sound of the chicken echoed all over. Nice!

During the day a lot of people were in and really sat down testing. There were also a lot of interesting collaboration going on between people that hadn’t met before. Some of the people that spent more time than others were Shmuel Gershon, Markus Gärtner, Isabel Evans, Michael Bolton, Ajay Balamurugadas, Zeger Van Hese, and Rob Sabourin (we might have forgot some of the names, but you know who you are).

During the day there were a lot of non-arranged  testing going on. But we had two really appreciated sessions during the day.
Firstly, there was a pair-testing session led by James Lyndsay where close to thirty people attended. There was a good mixture of experienced and less experienced testers. Even though the session was only about 30 minutes divided in three 10 minutes mini-sessions, it seemed to give the participants some very nice hands-on experience to bring home. And a lot of good collaboration was happening!

Secondly, during lunch hour it was an improvised weekend-testing session, Ajay Balamurugadas had invited as many as possible during his talk on Weekend Testing right before the session. Ajay and Markus Gärtner led the session with the rest of the participants that had showed up. Since there were so many that showed up we teamed up in pairs or smaller groups. The collaboration in the TestLab was yet again really great! After close to 30 minutes session we had found a bunch of interesting bugs while at the same time many have had the opportunity to team up with these fabulous  testers.

Despite that we had to cancel some planned sessions, we all thought that the first day was a real success . And we measured that by realizing that we didn’t have had time to take breaks or eat a proper lunch because the TestLab wasn’t empty for even a minute; there were at least 5 people there at minimum (except during the key note by Stuart Reid).

On the evening there was the Gala Awards dinner at Copenhagen City Hall and we ended up at a pub called Bryggeriet with the Rebel Alliance discussing testing in all forms. We went home pretty early and met up with some people in the hotel and continue discussing testing.

Thursday

The day kicked off at 08:00 in the TestLab and we used the first 30 minutes preparing for yet another interesting day!

At the AllSTAR testers, a few renown testers were able to join in. We split the TestLab into three groups where each focused on a specific area which they knew well. Each group attacked the systems in various ways, teaching the participants new ideas on how to test the systems. Ari Takanen was showing how he did some security testing (and/or fuzz testing); Rob Lambert showed how he did accessibility testing; and James Lyndsay had a exploratory testing session discussing a lot of test ideas with the participants.

At the round the table of managing ET, James Lyndsay moderated the group consisting of Rob Sabourin, Michael Bolton, Carsten Feilberg, Shmuel Gershon, Zeger Van Hese, Henrik Andersson, Henrik Emilsson, Martin Jansson, Markus Gärtner, John Stevenson, Neil Thompson and some more (we are sorry if we missed to mention someone, but we try our best to remember all people that were there out of those 200 TestLab attendees). We talked about several aspects around managing ET; and the discussion began with how we had managed our own time in the TestLab. James Lyndsay moderated the group in an excellent way. One of the subjects was how often you change/update your charters. Another interesting topic was on how much information you should provide different testers with.

During the lunch session, Markus Gärtner launched a Testing Dojo that was very appreciated. And Michael Bolton was doing a transcription of the session. The testing dojo meant that one tester tested the application with a projector that showed the computer screen. The tester then described through the testing and explaining what happened. All other around the tester could ask questions. The tester should be switched every 5 minutes letting several people show off their skills.
While this was the intention, the testing dojo slightly turned out to something different. But still as much appreciated by those involved.

Thursday evening we, Markus Gärtner, and Michael Bolton headed in to Copenhagen for a dinner with Ajay Balamurugadas, Teemu Vesala, Kalle Huttunen and Mika Hänninen. It took us two hours to get to the restaurant because we were forced to explore the Copenhagen Metro and train system…

Friday

We headed home to Sweden in the morning and arrived in Gothenburg at 15:00 in the afternoon. Then Henrik continued to Karlstad and arrived at 18:30. Seeing the problems other people had with getting out of Denmark by plane or train, we were lucky that car traffic wasn’t too affected by the blizzard…

Some numbers

• 180 people got in to the TestLab and got introduced to what we do (most of them sat down and tested)
• 100 hours of testing
• 100 reported bugs (plus several bugs not reported in the bug system)

Reflections

Considering how the TestLab was run, it was like a mix of having an open house and being a test lead handling several projects at the same time. We served different stakeholders such as the participants in the lab, the speakers, the conference personal and ourselves as apprentices as well as the masters (James and Bart). The plan is ever changing. Each stakeholder value different things. Just like in the real world outside the TestLab.

There were also 5 sponsor presentations from the tool vendors that had their tools on the client machines. We really appreciated those sponsors that had put in some time to really show how their tool could be used on the same applications as we tested in the TestLab. Very useful and meant that there were more value for those interested in the tools to try things out for themselves on the lab machines.

During the TestLab we got a lot of ideas on what we want to do next year. A lot of ideas that we had on beforehand were though only good in theory; seeing the real obstacles and problems in practice was a good lesson.

And finally, we really appreciate the great effort that James and Bart have done in creating the EuroSTAR TestLab and made it to such a remarkable success. We will have the benefit of building upon a very solid and well thought-out platform & concept!

We had a lot of fun and hope that all others had a great time!
See you all in Manchester 2011!

Cheers,
Henrik and Martin

The delegates of the first Swedish Workshop on Exploratory Testing were:
Torbjörn Ryber, Simon Morley, Christin Wiedemann, Petter Mattsson, Anders Claesson, Oscar Cosmo, Johan Hoberg, Rikard Edgren, Henrik Emilsson, Martin Jansson, Ann Flismark, Henrik Andersson, Michael Albrecht, Johan Jonasson, James Bach.
This write-up surely contains mistakes and important omissions, and it might be too heavy to read, but we bet you can jump in anywhere, read for a minute, and get something interesting out of it.

“15 people spending a weekend of their spare time to talk about Exploratory Testing!” (Andersson)
“This is the 1% of the 1%” (Bach)

Henrik Andersson – add Learning and Adaption to SBTM debriefing

Have experienced that testers stop doing debriefing after some time, which leads to lower Session Notes quality, or that they are skipped.
If it is a time issue, several debriefs can be made at once, or that testers debrief each other, or that all debriefs are made at once (30 minutes) at the end of the day.
The PROOF (Past, Results, Obstacles, Outlook, Feelings) mnemonic doesn’t support so much of coaching and learning, therefore Henrik advocates adding Learnings and Adaption (PROOF L.A.) that makes people discuss, learn, and use debriefs better (and stop skipping them.)

During debriefings you come up with new test ideas (Edgren)
Master the skill of note-taking
“the debrief formally accepts the session report” (Bach)
“The PROOF is (meta-)data regarding the session; LA is personal data (gathered from session)” (Emilsson)
“LA is not written down as a part of the debrief; it is only discussed orally” (Andersson)

The discussion went to many places, including status reports, and the need for training stakeholders if necessary.
“In my status reports, feelings are very important!” (Andersson)
“you almost need a handshake” (Morley)
Claesson shared a story from an East Asian company where the man-in-charge went to the most experienced troubleshooter and asked “Do you think the product is ready?”
“Troubleshooter as a career path in testing?” (Jansson)

debriefs can differ a lot depending on tester, area, type of work, experience etc.
a project manager should care about the team members’ long-term development
“testing is a learning experience in itself” (Claesson)
“don’t add things to models that already are good” (Mattsson)
“I love lists” (Wiedemann)
“do debriefs with a developer” (Flismark)
“break patterns to see what happens” (Andersson)
“are you satisfied with testers that don’t want/know how to learn?”

“it’s not a tool to fill the database” (Jansson)
As a single tester in a SCRUM team, you can have testers from different teams debrief each other.
“personal debriefing” is for the experienced
for inexperienced testers you don’t want to miss the coaching opportunity of debriefings
if you take away debriefs you might get too much freedom?
This area can be further analyzed and discussed, especially Group Debriefs.

Petter Mattsson – From 10.000 test cases to SBTM

At UIQ, producer of mobile phone software, they were running 10.000 manual test scripts, and did not find many bugs. These were instead found by customers…
Petter went to the Rapid Software Testing course and his conception of the world changed.
They decided to start using Exploratory Testing with Session-Based Test Management.
They got the opportunity to do a pilot that resulted in more, and more interesting, bugs.
A key to getting the Go decision, was a 2 hour workshop where all managers discussed questions like What is Quality? What is Testing?
They also did an exercise that showed the difference between scripted and exploratory testing.
The workshop was attended by all, the invitation was sent by a manager.

Michael Bolton did the RST course for all testers, that got inspired and motivated.
“RST with bonus day is great”
“we start with people that were hired two weeks ago, without testing experience”
They started with pair testing only, and then mixed with alone-testing.
Mixed different personality types, testers/developers, experienced/inexperienced.
“in the beginning you need to push them, but not after a while”
They developed a web-based solution for session report management (based on Perl tool)
They kept 500 test cases for verifying the basic functionality, and threw away the rest.
“we always need to do that mix” (of ET and ST)
They had really good results with the new approach, but a big phone comapny bought UIQ’s customer (that also was an owner), and the company was shut down.

Common transition problem: “ET is nice, but we have this problem, so we need to get back to the test cases.”
At another company, Petter has seen resistance to transition from testers, and also from managers, that don’t allow time to be spent.
“Just sit with them and test, let it grow organically”
“Why wait so many years for improving the way you work? Why wade in the mud when you can just step out?” (Jansson)
“They are not interested in testing; testers are used as a blame-shield” (Bach)
“do more of the robot dance” (Bach)
“If noone admits that there is a problem, noone owns the problem, and there is nothing to resolve” (Emilsson)
“ET helps you find important problems quickly”
“I’d like to have the testers off-site for a month, and just talk to them” (Mattsson)
“run the same test title, but without the steps”
heavy pain gives motivation
“They need to say: I am an alcoholic. My testing strategy really sucks” (Claesson)
“all organizations are perfectly designed to get what they get”
“small-scale change with one word: serendipity; scripted testing with deviations is a good start” (Edgren)
“I just removed the steps from the scripts, nobody noticed” (Cosmo)
Is RST the silver bullet? No, a money machine
“I’ll talk to one, that talk to others, and it spreads all around” (Mattsson)
It takes time to grow an exploratory testing team.
“If you say it will be hard, it will be hard, it’s a self-fulfilling prophecy” (Claesson)
“Improvisational Exploratory Scripted Testing” (Bach)
“What’s good in these test cases live on in your heart. You shouldn’t tell people to throw away something that is useful.” (Bach)
The scripted test become a ceremonial testing.

“it is the tester’s responsibility to improve” (Many)
“we should have a class: how to evangelize testing” (Bach)
You evangelize by giving real examples, from your own organization.
“the important thing is to talk about good testing” (Morley)
“It is not necessary to create a revolution, sometimes an evolution is better” (Jansson, Emilsson)
“motivation is the most important factor, a motivated team can achieve whatever they want” (Emilsson)
if nothing helps, you should use hard facts – bug stories, from real customers
take your most painful bugs, and talk about them.
FDA recalls often say “under certain circumstances”; that’s when you need Exploratory Testing (Bach)
http://www.fda.gov/MedicalDevices/Safety/RecallsCorrectionsRemovals/ListofRecalls/default.htm

Christine Wiedemann – starting with SBTM

They are an autonomous test group of 2 persons, that works with external customers.
Customer A – many requirements, a couple of hundred test cases, 3 weeks to execute.
Customer B,C,D – no requirements.

Customer A found showstopping bugs after production, and they realized:
“what we do doesn’t work; we have to do better testing”
In March/April they stopped writing test cases.
Inspired by SBTM, they started writing test execution notes in Word, sometimes working in pair.
“Now, we’re having fun when we are testing”, and they find the defects before production.
They are cooperating with developers and customers, and have even done ET workshops with the enemy (customers)
Customers are doing ET on the delivery, they have even taken the RST course!
“Management still doesn’t care what we do.”
We have a better understanding of the product and current high-risk areas.
Went from “trying to reach coverage” to “trying to find defects before production”
shared responsibility for quality

So what is left to do:
* more structured environment
* more documentation
* better time-boxing
* automation

“unfortunately, developers are too creative, and new features appear”
“I don’t need requirements. I love it.”
They use diagrams that describe the functionality (yEd)
they report bugs to the “ether”
“I have stopped being personally attached to bugs. Stopped arguing for bugs, instead telling developers ‘I saw something strange…’” (Ryber)
you can also say “this one is probably too difficult…”
Reality Steam Roller Method – let them go into suffering, but help them (Bach)
HICCUPPS – “I invented the name. I noticed people doing this.” (Bach)
Regarding more structured environment: “You are an explorer, explore a project and try to find out what might be important during a project’s lifecycle; and use this as a checklist for deciding on what to do and when.” (Emilsson)

granularity of Session notes may vary, richer reports have more information, but are harder to read.
“Sometimes I write down the humidity in the room” (Wiedemann)
Why do you want structure? Rather, be more aware of the structure.
“be the author, not the victim”
“unserendipity – to get through the test case without finding bugs” (Edgren)
“focus on what you want to achieve” (Albrecht)
“most important traits are awareness and willingness to learn” (Claesson)
trigger yourself for new test ideas
Anders Claesson shared a story that started as a tester learning about customer’s usage had ripple effects and evolved to a very good customer relation.

Ann Flismark – SBTM and KPI??

Went to STARWEST and got inspired.
Started with SBTM in December 2009, had problems integrating it in existing system, and are working on their own tool.
“what I prepared doesn’t make sense anymore”, so instead the focus was on requests for KPI, maybe they can help us as testers as well?

“Just say no” (All)
“Do testers feel productive?” (Edgren)
“How good do you think the product is?”
“tell the manager you will make something up” (Emilsson)
“you need to learn how to do status reporting” (Bach)
The true problem: how to deal with manager requests you don’t think are meaningful.

James Bach – Experience report involving Thread-Based Test Management

At a project, it was impossible to stick to completing the sessions.
There were many and continuous interruptions, and tasks could not be completed due to various reasons, “a bunch of pots boiling”.
The simple idea for this is: Thread-Based Test Management
“work backwards from the status report you want to be able to give”
“monitors status of test activities”
Why name it? So you can say “we switched from a session-based approach to thread-based.”
“This is such a simple idea. Too simple for a book, maybe a pamphlet.”
compare activity-based, artefact-based, and metrics-focused management

Is this the same as Kanban, without limiting work in progress?
No, key idea is “you acknowledge the fact that test activities rarely are finished” (Edgren, Emilsson)
In testing, you can’t always check off items on your To-do list (which (together with throughput) is the point of Kanban)
“the simple (and hard!) thing: think of things I’m working on” (Emilsson)
get out of the “Are you done yet?” trap
Waterfall and V-model might have started as jokes?
“project management tool focused on activities that rarely are finished”
Test Storytelling Tool (test management tools only handles test cases, not stories)
Are there any other professions where you want to transform activity results to status reports?
“don’t want to over-identify this until I have talked to you” (Bach)
which are areas of importance that need to be developed?

Cosmo uses a web forum for TBTM (thetesteye have used Word)
Mind Manager could be a tool, you can use icons to filter with. Tags or table columns for other tools.
“Do you feel you are trapped by SBTM or that SBTM is your silver bullet?” (Jansson)
James said he could get biased by SBTM, but “there are no silver bullets”, Fred Brooks
Common mistake: transform the map (model) to a list that should be Done (this doesn’t fit ongoing activities) (Emilsson)
There are many situations where it doesn’t matter if you can measure if you’re done.

“Testing doesn’t deliver anything except information about the product. Information that continuously emerges. Threads describe activities that lead to information that is part of our report” (Edgren)
There are (at least) two stories; one about the testing, and one about the product.
Story-based test management – threads are weaving into a report that is the story about the product
Test framing – thread design
Testing transforms a naive story about the product to a sophisticated product story. (Bach)
naive rumors -> testing -> empirically grounded information
different threads can be executed with one test activity
“twist together into a chord”
go out of the abstract world, and visualize testing
“everybody is already doing TBTM”
developer to a tester struck by the process: “don’t worry, we’re not gonna do anything they say anyway”
“you can all say ‘James Bach says I am a world expert in thread-based test management.’” (Bach)

Everybody was very happy with the peer conference, and let’s end by quoting Andersson’s check-out: “I’m pretty damn sure we have a bright future.”

/Rikard, Henrik, Martin

I thought of this the other day when rethinking a situation that was described in the experience report from Petter Mattson on “Swedish Workshop on Exploratory Testing” (SWET1).

Let’s say that you have 100 Paper Airplane builders in your team. They all follow scripted instructions on how to fold a paper in order to create a specific paper airplane; and the goal is to build airplanes that can fly.
But you notice a problem. None of the airplanes can fly! No matter of how much strength you use to throw them away, they instantly fall to the ground and crashes.

What you could do in this situation is to keep the scripted instructions and hope that mother nature changes so that the current airplanes can fly.
Or,  you could throw away the scripted instructions and let people use their minds, creativity, and exploratory skills in order to find out how a paper can be folded in order to get a functional paper airplane that can fly.

If what you are currently doing isn’t working, i.e. not fulfilling your goal (in this case to get an airplane to fly), a change cannot be more unsuccessful. It is instead an opportunity to get at least one airplane up and flying, even if it doesn’t look like one of the intended airplanes. Plunge in and each small progress is something that you will learn from.
I think that it is enough with just 1 paper airplane builder being successful and the other 99 just sitting down and having a cup of coffee. This is at least more successful than having 100 paper airplane builders not creating any value at all.

The Crashing Paper Airplane Heuristic: If you follow scripted instructions on how to build paper airplanes and you don’t create anything that can fly, throwing away those instructions and start building paper airplanes from scratch cannot be more unsuccessful.

When testing software, the very best inspiration and reality check comes from the software itself. It helps you test beyond requirements, and investigate what the software really is capable, and incapable, of.
These are my best practices for exploratory testing.

1. understand what’s important – we can’t test everything, and we can’t find all bugs. But it can be feasible to find all important bugs. This is very difficult, doesn’t involve a great deal of luck, but a good understanding of multiple, and diverse information sources. An Exploratory Testing trap is to spend a lot of time finding bugs that are interesting to the tester, but not to the project.
A key to important matters is to be subjective. I guarantee you, if your software is made for people, they will be subjective when they are using the product. Use your humanity and your feelings, they are worth more than the explicit requirements. The truth is the subjectivity (Kierkegaard)

2. be open to serendipity – the most important things are often found when looking for something else. If we knew exactly where all bugs were located, it would suffice with automation or detailed test scripts (or maybe we wouldn’t have to bother with tests at all?) So make sure that you look at the whole screen, and many other places as well. Odd behavior might be irrelevant, or a clue to very important information. Testing always involves sampling, and serendipity can be your friend and rescue.

3. work hard, but don’t – manual, exploratory testing can give enormous coverage in a short time when pieces are in the right place. At the same time, you should pause your fast runs and look from different perspectives, both in order to see more things, but also to not get too tired. Sometimes you need to investigate details for a long time without progress, sometimes you need to do something completely different to get fresh eyes. Stay focused, lose focus. Exploratory Testing Dynamics is a profound document with lots on this.
Structure and discipline are key parts of any successful testing effort, but also to combine disparate information sources (often called creativity.)

4. work close with developers – in physical location, and in time (tight feedback loop: code-test-fix-verify). Talk to them instead of reporting a vague bug; earn their trust, and share information.
But testers should think differently, if everybody looked at the software in the same way, we wouldn’t get the broad coverage it (hopefully) will be exposed to after release.
It is good to work with other roles as well, but developer collaboration is the most important.

5. one-liner test ideas – if you lightly document the test ideas you intend to execute, you can give ideas to developers even before they have written the code (with the inevitable bugs.) You can also get feedback on importance and improvements by fellow testers, other interested and stakeholders. With appropriate granularity you can go below 100 test ideas, and make it possible for others to understand the testing that will be performed, and comment on what is missing.

The list could be made longer (broad learning, work in other roles, CRUSCPIC STMPLA, see the big picture, SBTM, creativity…), but I think it’s better to focus on a few important. Who knows, I might be totally wrong, and don’t want to spend too much of your precious time.

-

* There is no such thing as something absolutely, objectively best. This goes for all situations.
An Olympic champion was the best at that moment according to the rules that were used in that competition.
Newton’s physics were the best for a while, and are still good for approximations in real life.
“The Beatles are the best band in history” is a subjective statement, shared by many.
So in this article, feel free to exchange “Best Practice” with “good examples for me that I hope are inspiring”.
Ask Wittgenstein, language is a game. You decide what to do with the words.

Let me begin by a quote from T.S. Eliot:

We shall not cease from exploration, and the end of all our exploring will be to arrive where we started and know the place for the first time.

One of the most important things with Exploratory Testing is that it allows for you to learn something during the exploration. It is urging you to think for yourself; draw conclusions; misunderstand; learn from mistakes; see the full picture; noticing details.
Learning about the product and the context is a key element in discovering problems that might bug users and stakeholders.

I have a problem with remembering and learning stuff if I (or someone else) have written something down and I’m using the text as the answer each time. E.g. a set of instructions on how to perform a special action; a login password; etc. But if I have done the thing myself, I have no difficulties in performing the special action. And perhaps most importantly, I have an understanding of every needed step and why each step is needed.
When I am asked to review something, it is important that I explore the area in order to learn something from it. To me, this is (almost) the only way to see what is missing. I can do a good job by finding problems in the things that are already in there, but in order to see what is missing I have to learn about the area and what it really means.

It is the same thing with Scripted Testing vs. Exploratory Testing. If tests are scripted, there is no need to reflect upon what you do. The script is there so you don’t have to think. An exploration, on the other hand, requires thought work. It requires that your brain is in an alert state and that you learn something from what you perceive. ***

I also think that the learning part of exploratory testing is what matters when comparing two testers with similar resumes. Ask the candidates if they have had an exploratory approach in the reference projects, and ask them what they have learned.
I bet that you will spot those who have learned something and thereby improved their testing skills, including how to think about the context and its affection. I also believe that they have improved their problem solving skills and boosted their creativity along the way.

Do you have any learning experiences that you can share with us?
Have you experienced when you didn’t learn something? What did that do to you?

*** Note: Of course it depends on where you are on the Scripted vs. Exploratory continuum.