A year has passed since we released version 1.0 of our quality model without metrics.
It is time for a new version, with additions and corrections we have learned over the year (and a Swedish translation!)

1.1 English
1.1 Swedish
1.0 English

Feedback is always welcome!

/Rikard, Henrik, Martin

The delegates of the second Swedish Workshop on Exploratory Testing (Test Planning and Status Reporting for Exploratory Testing) were:
Henrik Andersson, Azin Bergman, Robert Bergqvist, Sigge Birgisson, Rikard Edgren, Henrik Emilsson, Ola Hyltén, Martin Jansson, Johan Jonasson, Saam Koroorian, Simon Morley, Torbjörn Ryber, Fredrik Scheja, Christin Wiedemann, Steve Öberg

Discussions on peer conferences can’t be summarized, but you can read the abstracts, Ryber’s notes, and here comes some quote highlights:

- How many test cases do you want? 800? Then let’s say we have 800. (Jonasson)
- Think “there are problems”, not “there might be problems” (Bergqvist)
- To motivate: lead by example, avoid de-motivating (Hyltén, Andersson)
- They want to see bug curves, but don’t know it going up or down is good or bad. (Jonasson)
- Rather ask “What are you afraid of?” (Jansson)
- Qualitative information is sensitive to filtering (Emilsson)
- Dialogue is important! (Morley)
- Exploratory test planning, adapt to reality. That’s how we all work. (Scheja)
- A test expert that doesn’t know anything (favorite slip of the tongue from Jansson)
- Has this story confirmed that anyone can test? (Bergman)
- Incorrect expectations: if we test, there won’t be any production problem (Ryber)
- Categorizations are good for learning, but throw them away to meet reality (Edgren)
- I always use low-tech testing dashboards, except in this case (Ryber)
- A tester is also test leader, test designer, test planner (Scheja)
- It is small raisins in a very large cake (Jonasson)
- I want to test until the world is on fire (Wiedemann)
- Care about what is most important (Emilsson)
- I invent many wheels every day (Scheja)
- I want more emphasis on which information we have and don’t have, and the confidence of the information (Koroorian)
- Beware of using the word “coverage”, rather say security, or risk (Birgisson)
- To me, coverage is a start of thinking about what to test (Bergqvist)
- If this isn’t reported “upwards”, you avoid the biggest risks (Andersson)
- The map and the creation of it is more important than the numbers on it (Öberg)
- It might look easy with a number, but it isn’t (Edgren)
- This weekend is evidence that standards don’t work. We can discuss forever, and that’s what is fun! (Emilsson)

These Lightning Talks (5 minutes including questions) were held:
* Martin Jansson on the tester’s greatest nemesis, the view that anyone, thus any idiot, can test.
* Henrik Emilsson on unjustified tests that open up to serendipity and new ideas.
* Robert Bergqvist had an experience report that showed that exploratory testing also needs planning.
* Azin Bergman on the common view that testers have lower status than other roles.
* Simon Morley on root cause analysis heuristic FICL: Framing, Information Gathering, Consensus, Learning.
* Rikard Edgren on Binary Disease – our tools have shaped way too computeresque theories.
* Christin Wiedemann on open-ended plans without fixed scope.
* Sigge Birgisson shared an experience report with developer collaboration under the radar.
* Steve Öberg led a brainstorm on testing analogies with story from the Aeneid about not following the Delphi Oracle.
* Ola Hyltén showed the Johari window that can enable better communication between leaders and team members.
* Torbjörn Ryber led a discussion about doing a context-driven conference in Sweden.

It is very easy to organize a workshop like this when you have 15 motivated, passionate testers.
You just need a venue with few distractions, a theme to focus, an agenda and discussion rules, plus food and drinks.

Looking forward to SWET3!

Monday

Henrik started his journey by car from Karlstad and went down to Gothenburg to join with Martin. Both of us were going to take the train down to Copenhagen. Not surprisingly there were delays and the train was cancelled… Instead we headed back to Martin’s house and loaded Henrik’s car and took off. After several hours drive we arrived to Copenhagen, where we went directly to Bella Center and the TestLab at EuroSTAR.

This was the first time we met James Lyndsay and Bart Knaack in person; we had only talked with them over Skype/phone. So we all started with saying hello but there were really no time to sit down and have a cup of coffee – we needed to get to work straight away.

One of the burning issues to resolve the week before the conference was to bring client machines. Henrik was able to bring four laptops (thank you Compare Testlab and Know IT for sponsoring with this!) and Martin one. We worked in the TestLab for a few hours setting up and testing the two networks (James had one and Bart had another) with the servers connected to them.

Each network consisted of a bug tracker (Mantis), a server with OpenEMR, a server with a WebShop based on OsCommerce as well as a link to download a version of FreeMind.

When the conference center was getting near closing hours we rounded up and identified tasks needed to be done for the next coming days. We were assigned to setup and fill both the OpenEMR and the WebShop with new test data. We determined that we first should create a data model in FreeMind; both for reviewing before inserting the data and also so that we could print out the model and use it as a means to discuss with participants in the TestLab. With the model at hand, we thought the participants would also be able to identify test ideas of their own that they could try out.

We packed up everything and took Henriks car together with James and Bart. Half an hour later we were down in the hotel bar for dinner. We first met up with Steve Öberg, Rikard Edgren, Björn Karlsson, and Stefan Thoresson; and later on Markus Gärtner, Neil Thompson & Michael Bolton and some more joined our table and we were discussing testing long into the night.

Tuesday

At 8.45 we met up with James and Bart at the hotel and took a cab to the conference center where we setup the TestLab again. We started on the data models while James and Bart worked on setting up the details of the two networks. Meanwhile the sponsors were installing their applications on the client machines. Late into the afternoon we were starting to get ready with the new data for both systems. During the afternoon several people were visiting us and some of them started testing the applications. In fact, the first bug entered in the bug system was the one that won the prize “Best bug” – Markus Gärtner found it pretty fast…
Around 18:00 we were able to move into the room where the TestLab should be. We moved in all servers, routers and clients and setup the power for it all. Funny thing is that we didn’t find more than one electricity socket, so all machines were hooked up on this only one and we decided to load test the electricity over night.
During the evening we attended the Rebel Alliance (or Danish Alliance) for 11 interesting lightning talks. The discussions went on after that long into the night. But to our defense we must say that we got in bed in time.

Wednesday

Despite that we got in bed fairly early, we both overslept and missed the grand opening of the TestLab… Such a good start!
But no time to cry over that; we dug in and assisted in welcoming and introducing people that entered the TestLab and told them about it and how to use it; as well as guiding them to start testing. Bugs were reported and the sound of the chicken echoed all over. Nice!

During the day a lot of people were in and really sat down testing. There were also a lot of interesting collaboration going on between people that hadn’t met before. Some of the people that spent more time than others were Shmuel Gershon, Markus Gärtner, Isabel Evans, Michael Bolton, Ajay Balamurugadas, Zeger Van Hese, and Rob Sabourin (we might have forgot some of the names, but you know who you are).

During the day there were a lot of non-arranged  testing going on. But we had two really appreciated sessions during the day.
Firstly, there was a pair-testing session led by James Lyndsay where close to thirty people attended. There was a good mixture of experienced and less experienced testers. Even though the session was only about 30 minutes divided in three 10 minutes mini-sessions, it seemed to give the participants some very nice hands-on experience to bring home. And a lot of good collaboration was happening!

Secondly, during lunch hour it was an improvised weekend-testing session, Ajay Balamurugadas had invited as many as possible during his talk on Weekend Testing right before the session. Ajay and Markus Gärtner led the session with the rest of the participants that had showed up. Since there were so many that showed up we teamed up in pairs or smaller groups. The collaboration in the TestLab was yet again really great! After close to 30 minutes session we had found a bunch of interesting bugs while at the same time many have had the opportunity to team up with these fabulous  testers.

Despite that we had to cancel some planned sessions, we all thought that the first day was a real success . And we measured that by realizing that we didn’t have had time to take breaks or eat a proper lunch because the TestLab wasn’t empty for even a minute; there were at least 5 people there at minimum (except during the key note by Stuart Reid).

On the evening there was the Gala Awards dinner at Copenhagen City Hall and we ended up at a pub called Bryggeriet with the Rebel Alliance discussing testing in all forms. We went home pretty early and met up with some people in the hotel and continue discussing testing.

Thursday

The day kicked off at 08:00 in the TestLab and we used the first 30 minutes preparing for yet another interesting day!

At the AllSTAR testers, a few renown testers were able to join in. We split the TestLab into three groups where each focused on a specific area which they knew well. Each group attacked the systems in various ways, teaching the participants new ideas on how to test the systems. Ari Takanen was showing how he did some security testing (and/or fuzz testing); Rob Lambert showed how he did accessibility testing; and James Lyndsay had a exploratory testing session discussing a lot of test ideas with the participants.

At the round the table of managing ET, James Lyndsay moderated the group consisting of Rob Sabourin, Michael Bolton, Carsten Feilberg, Shmuel Gershon, Zeger Van Hese, Henrik Andersson, Henrik Emilsson, Martin Jansson, Markus Gärtner, John Stevenson, Neil Thompson and some more (we are sorry if we missed to mention someone, but we try our best to remember all people that were there out of those 200 TestLab attendees). We talked about several aspects around managing ET; and the discussion began with how we had managed our own time in the TestLab. James Lyndsay moderated the group in an excellent way. One of the subjects was how often you change/update your charters. Another interesting topic was on how much information you should provide different testers with.

During the lunch session, Markus Gärtner launched a Testing Dojo that was very appreciated. And Michael Bolton was doing a transcription of the session. The testing dojo meant that one tester tested the application with a projector that showed the computer screen. The tester then described through the testing and explaining what happened. All other around the tester could ask questions. The tester should be switched every 5 minutes letting several people show off their skills.
While this was the intention, the testing dojo slightly turned out to something different. But still as much appreciated by those involved.

Thursday evening we, Markus Gärtner, and Michael Bolton headed in to Copenhagen for a dinner with Ajay Balamurugadas, Teemu Vesala, Kalle Huttunen and Mika Hänninen. It took us two hours to get to the restaurant because we were forced to explore the Copenhagen Metro and train system…

Friday

We headed home to Sweden in the morning and arrived in Gothenburg at 15:00 in the afternoon. Then Henrik continued to Karlstad and arrived at 18:30. Seeing the problems other people had with getting out of Denmark by plane or train, we were lucky that car traffic wasn’t too affected by the blizzard…

Some numbers

• 180 people got in to the TestLab and got introduced to what we do (most of them sat down and tested)
• 100 hours of testing
• 100 reported bugs (plus several bugs not reported in the bug system)

Reflections

Considering how the TestLab was run, it was like a mix of having an open house and being a test lead handling several projects at the same time. We served different stakeholders such as the participants in the lab, the speakers, the conference personal and ourselves as apprentices as well as the masters (James and Bart). The plan is ever changing. Each stakeholder value different things. Just like in the real world outside the TestLab.

There were also 5 sponsor presentations from the tool vendors that had their tools on the client machines. We really appreciated those sponsors that had put in some time to really show how their tool could be used on the same applications as we tested in the TestLab. Very useful and meant that there were more value for those interested in the tools to try things out for themselves on the lab machines.

During the TestLab we got a lot of ideas on what we want to do next year. A lot of ideas that we had on beforehand were though only good in theory; seeing the real obstacles and problems in practice was a good lesson.

And finally, we really appreciate the great effort that James and Bart have done in creating the EuroSTAR TestLab and made it to such a remarkable success. We will have the benefit of building upon a very solid and well thought-out platform & concept!

We had a lot of fun and hope that all others had a great time!
See you all in Manchester 2011!

Cheers,
Henrik and Martin

With all due respect, this is the announcement of the perhaps most powerful public two-page document in the history of software testing.
It is an extended re-write of James Bach’s Quality Criteria Categories, and has been developed to 12 categories (CRUCSPIC STMP) and 93 sub-categories for software quality characteristics/attributes/factors/dimensions/properties/criteria/aspects.
This list is not objectively true, and not easy to use for measuring (compare with ISO 9126-1), but you can adapt it to your context and be inspired by it when understanding/creating/reviewing software quality related stuff.

The PDF is available here, and goes under the Creative Commons Attribution-NoDerivates license, as all our Publications.
Use the list to generate test ideas, or to discuss what is important; get inspired by the concept of Charisma (SPACE HEADS mnemonic)
Suggestions for improvements to the list are very welcome!

We will continuously blog about some of the categories (and sub-categories) in order to provide more information and examples of usage.

Rikard, Henrik, Martin
the test eye

The delegates of the first Swedish Workshop on Exploratory Testing were:
Torbjörn Ryber, Simon Morley, Christin Wiedemann, Petter Mattsson, Anders Claesson, Oscar Cosmo, Johan Hoberg, Rikard Edgren, Henrik Emilsson, Martin Jansson, Ann Flismark, Henrik Andersson, Michael Albrecht, Johan Jonasson, James Bach.
This write-up surely contains mistakes and important omissions, and it might be too heavy to read, but we bet you can jump in anywhere, read for a minute, and get something interesting out of it.

“15 people spending a weekend of their spare time to talk about Exploratory Testing!” (Andersson)
“This is the 1% of the 1%” (Bach)

Henrik Andersson – add Learning and Adaption to SBTM debriefing

Have experienced that testers stop doing debriefing after some time, which leads to lower Session Notes quality, or that they are skipped.
If it is a time issue, several debriefs can be made at once, or that testers debrief each other, or that all debriefs are made at once (30 minutes) at the end of the day.
The PROOF (Past, Results, Obstacles, Outlook, Feelings) mnemonic doesn’t support so much of coaching and learning, therefore Henrik advocates adding Learnings and Adaption (PROOF L.A.) that makes people discuss, learn, and use debriefs better (and stop skipping them.)

During debriefings you come up with new test ideas (Edgren)
Master the skill of note-taking
“the debrief formally accepts the session report” (Bach)
“The PROOF is (meta-)data regarding the session; LA is personal data (gathered from session)” (Emilsson)
“LA is not written down as a part of the debrief; it is only discussed orally” (Andersson)

The discussion went to many places, including status reports, and the need for training stakeholders if necessary.
“In my status reports, feelings are very important!” (Andersson)
“you almost need a handshake” (Morley)
Claesson shared a story from an East Asian company where the man-in-charge went to the most experienced troubleshooter and asked “Do you think the product is ready?”
“Troubleshooter as a career path in testing?” (Jansson)

debriefs can differ a lot depending on tester, area, type of work, experience etc.
a project manager should care about the team members’ long-term development
“testing is a learning experience in itself” (Claesson)
“don’t add things to models that already are good” (Mattsson)
“I love lists” (Wiedemann)
“do debriefs with a developer” (Flismark)
“break patterns to see what happens” (Andersson)
“are you satisfied with testers that don’t want/know how to learn?”

“it’s not a tool to fill the database” (Jansson)
As a single tester in a SCRUM team, you can have testers from different teams debrief each other.
“personal debriefing” is for the experienced
for inexperienced testers you don’t want to miss the coaching opportunity of debriefings
if you take away debriefs you might get too much freedom?
This area can be further analyzed and discussed, especially Group Debriefs.

Petter Mattsson – From 10.000 test cases to SBTM

At UIQ, producer of mobile phone software, they were running 10.000 manual test scripts, and did not find many bugs. These were instead found by customers…
Petter went to the Rapid Software Testing course and his conception of the world changed.
They decided to start using Exploratory Testing with Session-Based Test Management.
They got the opportunity to do a pilot that resulted in more, and more interesting, bugs.
A key to getting the Go decision, was a 2 hour workshop where all managers discussed questions like What is Quality? What is Testing?
They also did an exercise that showed the difference between scripted and exploratory testing.
The workshop was attended by all, the invitation was sent by a manager.

Michael Bolton did the RST course for all testers, that got inspired and motivated.
“RST with bonus day is great”
“we start with people that were hired two weeks ago, without testing experience”
They started with pair testing only, and then mixed with alone-testing.
Mixed different personality types, testers/developers, experienced/inexperienced.
“in the beginning you need to push them, but not after a while”
They developed a web-based solution for session report management (based on Perl tool)
They kept 500 test cases for verifying the basic functionality, and threw away the rest.
“we always need to do that mix” (of ET and ST)
They had really good results with the new approach, but a big phone comapny bought UIQ’s customer (that also was an owner), and the company was shut down.

Common transition problem: “ET is nice, but we have this problem, so we need to get back to the test cases.”
At another company, Petter has seen resistance to transition from testers, and also from managers, that don’t allow time to be spent.
“Just sit with them and test, let it grow organically”
“Why wait so many years for improving the way you work? Why wade in the mud when you can just step out?” (Jansson)
“They are not interested in testing; testers are used as a blame-shield” (Bach)
“do more of the robot dance” (Bach)
“If noone admits that there is a problem, noone owns the problem, and there is nothing to resolve” (Emilsson)
“ET helps you find important problems quickly”
“I’d like to have the testers off-site for a month, and just talk to them” (Mattsson)
“run the same test title, but without the steps”
heavy pain gives motivation
“They need to say: I am an alcoholic. My testing strategy really sucks” (Claesson)
“all organizations are perfectly designed to get what they get”
“small-scale change with one word: serendipity; scripted testing with deviations is a good start” (Edgren)
“I just removed the steps from the scripts, nobody noticed” (Cosmo)
Is RST the silver bullet? No, a money machine
“I’ll talk to one, that talk to others, and it spreads all around” (Mattsson)
It takes time to grow an exploratory testing team.
“If you say it will be hard, it will be hard, it’s a self-fulfilling prophecy” (Claesson)
“Improvisational Exploratory Scripted Testing” (Bach)
“What’s good in these test cases live on in your heart. You shouldn’t tell people to throw away something that is useful.” (Bach)
The scripted test become a ceremonial testing.

“it is the tester’s responsibility to improve” (Many)
“we should have a class: how to evangelize testing” (Bach)
You evangelize by giving real examples, from your own organization.
“the important thing is to talk about good testing” (Morley)
“It is not necessary to create a revolution, sometimes an evolution is better” (Jansson, Emilsson)
“motivation is the most important factor, a motivated team can achieve whatever they want” (Emilsson)
if nothing helps, you should use hard facts – bug stories, from real customers
take your most painful bugs, and talk about them.
FDA recalls often say “under certain circumstances”; that’s when you need Exploratory Testing (Bach)
http://www.fda.gov/MedicalDevices/Safety/RecallsCorrectionsRemovals/ListofRecalls/default.htm

Christine Wiedemann – starting with SBTM

They are an autonomous test group of 2 persons, that works with external customers.
Customer A – many requirements, a couple of hundred test cases, 3 weeks to execute.
Customer B,C,D – no requirements.

Customer A found showstopping bugs after production, and they realized:
“what we do doesn’t work; we have to do better testing”
In March/April they stopped writing test cases.
Inspired by SBTM, they started writing test execution notes in Word, sometimes working in pair.
“Now, we’re having fun when we are testing”, and they find the defects before production.
They are cooperating with developers and customers, and have even done ET workshops with the enemy (customers)
Customers are doing ET on the delivery, they have even taken the RST course!
“Management still doesn’t care what we do.”
We have a better understanding of the product and current high-risk areas.
Went from “trying to reach coverage” to “trying to find defects before production”
shared responsibility for quality

So what is left to do:
* more structured environment
* more documentation
* better time-boxing
* automation

“unfortunately, developers are too creative, and new features appear”
“I don’t need requirements. I love it.”
They use diagrams that describe the functionality (yEd)
they report bugs to the “ether”
“I have stopped being personally attached to bugs. Stopped arguing for bugs, instead telling developers ‘I saw something strange…’” (Ryber)
you can also say “this one is probably too difficult…”
Reality Steam Roller Method – let them go into suffering, but help them (Bach)
HICCUPPS – “I invented the name. I noticed people doing this.” (Bach)
Regarding more structured environment: “You are an explorer, explore a project and try to find out what might be important during a project’s lifecycle; and use this as a checklist for deciding on what to do and when.” (Emilsson)

granularity of Session notes may vary, richer reports have more information, but are harder to read.
“Sometimes I write down the humidity in the room” (Wiedemann)
Why do you want structure? Rather, be more aware of the structure.
“be the author, not the victim”
“unserendipity – to get through the test case without finding bugs” (Edgren)
“focus on what you want to achieve” (Albrecht)
“most important traits are awareness and willingness to learn” (Claesson)
trigger yourself for new test ideas
Anders Claesson shared a story that started as a tester learning about customer’s usage had ripple effects and evolved to a very good customer relation.

Ann Flismark – SBTM and KPI??

Went to STARWEST and got inspired.
Started with SBTM in December 2009, had problems integrating it in existing system, and are working on their own tool.
“what I prepared doesn’t make sense anymore”, so instead the focus was on requests for KPI, maybe they can help us as testers as well?

“Just say no” (All)
“Do testers feel productive?” (Edgren)
“How good do you think the product is?”
“tell the manager you will make something up” (Emilsson)
“you need to learn how to do status reporting” (Bach)
The true problem: how to deal with manager requests you don’t think are meaningful.

James Bach – Experience report involving Thread-Based Test Management

At a project, it was impossible to stick to completing the sessions.
There were many and continuous interruptions, and tasks could not be completed due to various reasons, “a bunch of pots boiling”.
The simple idea for this is: Thread-Based Test Management
“work backwards from the status report you want to be able to give”
“monitors status of test activities”
Why name it? So you can say “we switched from a session-based approach to thread-based.”
“This is such a simple idea. Too simple for a book, maybe a pamphlet.”
compare activity-based, artefact-based, and metrics-focused management

Is this the same as Kanban, without limiting work in progress?
No, key idea is “you acknowledge the fact that test activities rarely are finished” (Edgren, Emilsson)
In testing, you can’t always check off items on your To-do list (which (together with throughput) is the point of Kanban)
“the simple (and hard!) thing: think of things I’m working on” (Emilsson)
get out of the “Are you done yet?” trap
Waterfall and V-model might have started as jokes?
“project management tool focused on activities that rarely are finished”
Test Storytelling Tool (test management tools only handles test cases, not stories)
Are there any other professions where you want to transform activity results to status reports?
“don’t want to over-identify this until I have talked to you” (Bach)
which are areas of importance that need to be developed?

Cosmo uses a web forum for TBTM (thetesteye have used Word)
Mind Manager could be a tool, you can use icons to filter with. Tags or table columns for other tools.
“Do you feel you are trapped by SBTM or that SBTM is your silver bullet?” (Jansson)
James said he could get biased by SBTM, but “there are no silver bullets”, Fred Brooks
Common mistake: transform the map (model) to a list that should be Done (this doesn’t fit ongoing activities) (Emilsson)
There are many situations where it doesn’t matter if you can measure if you’re done.

“Testing doesn’t deliver anything except information about the product. Information that continuously emerges. Threads describe activities that lead to information that is part of our report” (Edgren)
There are (at least) two stories; one about the testing, and one about the product.
Story-based test management – threads are weaving into a report that is the story about the product
Test framing – thread design
Testing transforms a naive story about the product to a sophisticated product story. (Bach)
naive rumors -> testing -> empirically grounded information
different threads can be executed with one test activity
“twist together into a chord”
go out of the abstract world, and visualize testing
“everybody is already doing TBTM”
developer to a tester struck by the process: “don’t worry, we’re not gonna do anything they say anyway”
“you can all say ‘James Bach says I am a world expert in thread-based test management.’” (Bach)

Everybody was very happy with the peer conference, and let’s end by quoting Andersson’s check-out: “I’m pretty damn sure we have a bright future.”

/Rikard, Henrik, Martin

Today we celebrate our 100th post on this blog!

It has been an interesting journey for us so far; and we realize that we have only begun this ride, a ride with no destination but to enrich ourselves with wisdom and knowledge through discussions and by sharing thoughts. And you, our readers, are a very important part of this process.

Our decision to write posts in English has been a good strategy as we see it; not only do we reach people all over the world, but we can also discuss matters with peers that, in this community, don’t have any country borders. But you have to bear with us sometimes if you think that our English isn’t perfect, we are working on it… We do hope that the message gets through though.

The thing that we are three persons sharing a passion for testing and a blog to express it, has been a really good thing as we see it. Not only is it more fun, but it is also some assurance that the blog will keep up a good tempo; since there is always someone of us that has inspiration and time to write. And it will also be a guarantee that the topics that we choose to write about are evolving and often diversified. Over the years we have noticed that we agree on many things, but disagree or have different perspective on as many.

We are very happy to see that we now have a really large group of readers from all over the world, and we have had a huge increase of new readers over the last two years. We hope that you will keep finding us interesting and continue reading in the future; and we invite you to contribute even more with your comments on our thoughts.

Do you have any thoughts on where you think we should go from here on?

Best regards,

Henrik, Martin & Rikard
the test eye

Statements in specifications try to clarify and are inevitably an interpretation of what the author thinks need to be more specific. I.e., they try to be a more specific model than what existed before the spec. And “Essentially, all models are wrong, but some are useful” (http://en.wikiquote.org/wiki/George_E._P._Box).

Every specification you encounter is persons’ interpretations, and  not necessarily true.

This means that you as an inquisitive tester have a lot to do by questioning the specifications. The questioning will help you to form a model of the software that is better than if you only had read and accepted the spec as it was.

Specifications cannot be complete and especially regarding things that the program shouldn’t do. It is probably not stated that the software shouldn’t use too much memory or processor for certain operations; it is not stated that the screen shouldn’t flicker, or that all text should be easy to read with all different font settings. Other typical omissions are interactions with other systems; things you expect from all applications under that operating system, internet browser, connected software etc.

You cannot expect a specification to be complete, in most (all? many?) cases, the thing produced by the specification is more important than the document about it. The hardest challenge for the inquisitive tester is to question a lot, but only for those things that are important.

——————–

Who will use the specification? What will they use it for? Will it meet their requirements?

What is it all about? Really?

What areas are left out?

Who is the writer? Does he/she usually miss certain things?

Are there many writers? Does this make the whole less tangible?

Are there many reviewers? Are they using different perspectives?

Is the writer vague, insecure and confusing about certain areas?

Is the specification consistent?

Is the specification consistent with other related specifications?

Is the specification consistent with other different features and combinations of those?

Are all functional and non-functional requirements covered?

Are there dubious thoughts about the wished for functionality?

Are there other sources of information that can be useful?

How is the style of the language affecting the specification?

What quality attributes are the most important, e.g. how is Security weighed against Performance and Usability?

Does it match the system requirements?

Does the specification focus on what is most important?

Does the specification reflect the model of what you think is described?

Are there any new terminology? Will this affect other documentation such as help files?

Is the new terminology consistent with other specifications?

What does the Internet say about the newly chosen terminology? Will there be any misunderstandings?

—————–

If there was no specification, could it be described in a completely different way?

In order to become a successful inquisitive tester, there are a couple of things you can do to improve your skills beyond the more common quest to “question a product”. One important thing is to question the tests themselves.

——————–

Have you ever run tests and wondered if they were really necessary, perhaps knowing that the tests are useless?
Have you ever run tests that were too old and not updated with latest terminology or functionality?
Do your tests contain too much configuration information? Should this be put elsewhere?
Have tests become redundant because those failures no longer happen, ever?
Have the intent of the test been lost and therefore been rendered useless?
Have the tests already been run, on the same build? Twice?
Have you found any bugs or any important information at all with the tests?

Are your tests the most powerful?
Are the tests credible?
Can the tests be faster to execute?
Can you run the most important tests first?
Are the tests too narrow and/or too general?
Do you really understand the test?
Have the original test idea been lost in translation?
Are the tests too much of a projection of the test designer’s thought of view?

Are your tests interesting or boring to execute?
Are the tests in line with your test strategies?
How often do you change your test approaches?
Can the tests instead be better used as input to developers’ unit tests?
Can the essence of the tests be used elsewhere?
Have your tests been reviewed by your colleagues, including technical writers?
Have your scenario tests been reviewed by business people?
Have you captured how the users will use the software in the tests?

Are you satisfied with your tests?
Are your (hidden) stakeholders satisfied with your tests?

——————–

Can you come up with more questions?
Regards,
the test eye (Henrik, Martin & Rikard)