On 7-9 May the Let’s Test Conference will take place. During the day there will be lots of interesting tutorials, keynotes and sessions. During the evening the events will continue. One of these activities is the Testlab, that we call Let’s TestLab. Initially I misunderstood Henrik Emilsson when we started to organise the lab. I thought the evening event was the testlab. At the time I did not consider anything wrong with having 150 people or more in the testlab. As I saw the evening event program I considered how could I compete with such a fine setup of activities. Well, this is a conference with many context-driven people. We will have different interests, expectations and focus area. So instead of admiting defeat, I considered what elements that we could add to the testlab to bring a great crowd, but perhaps not all of them.

Here is the line up of events in the testlab:

• Collaborative test planning
• Group Test Experiment
• Test Competition
• Making a compelling status report

Collaborative test planning

We will create possible charters, missions for the coming testing in the lab so that those who wish can practise different testing techniques. Everyone is invited to share his or her idea on how to plan testing in through collaboration.

Part of the line up in this event is:

• Rikard Edgren
• Christin Wiedemann

Group Test Experiment

The context of the testlab will be single testers or groups of testers going in and out of the testlab in different time intervals. Each tester will be unique in the sense that they bring different level of experience, skills and approaches to testing. Based on this, we will start experimenting with group testing. I think we have limited ourselves too long with the setup of pair testing. Going back to the early recommendations and experiences from Brian Marick, James Bach, Cem Kaner, Jonathan Kohl and many others, the setup is nearly the same. With new tools and techniques appearing over the years, some assumptions could be questioned.

Let us assume that there are different aspects and combinations of group testing that serves different purposes, we can say that there are different dimensions that could be explored.

Here are a few dimensions that we will experimented with:

• How many testers (2 or more)
• What role you play as a tester
• User types, User Scenarios or storytelling
• Mission of group test
• Note taking techniques
• Partner combination
• Lateral thinking aspects
• Personality types
• Debriefing techniques
• Accountability
• Focus areas or Characteristic focus
• Test Environment
• Basic Configuration Matrix
• … and new ones that we find along the way

We will explore and experiment with different tools that we use when group testing and share experience on what works in what context. We also experiment with a few pre-defined group test setups such as:

• Cinema testing
• 15-min test run
• Coaching a group of testers
• Wolf pack concept
• Testing Dojo

Part of the line up in the test lab is:

• Ann-Marie Charrett
• Markus Gärtner and Meke Mertsch
• Johan Åtting

Test Competition

Can you really compete in testing? Can you compete between two or more teams?  Can you really estimate the value of one piece of information against another? Well, it depends.

I have been an Ultimate Frisbee player for 34 years. I’ve not played for some time, but once a frisbee fan, always a fan. I think the same goes for testing. There are many things that I feel is similar. Craftmanship/sportmanship and passion is major part. Here is one description I like:

Ultimate has traditionally relied upon a spirit of sportsmanship which places the responsibility for fair play on the player. Highly competitive play is encouraged, but never at the expense of the bond of mutual respect between players, adherence to the agreed upon rules of the game, or the basic joy of play. Protection of these vital elements serves to eliminate adverse conduct from the Ultimate field. Such actions as taunting of opposing players, dangerous aggression, intentional fouling, or other “win-at-all-costs” behavior are contrary to the spirit of the game and must be avoided by all players.

One important element in Ultimate Frisbee is the spirit of the game, which you can see more in detail here [1]. Passion and humility as a tester are important traits, the spirit of the game concept might help us here.

So, my idea for a tester competition will be based on some of these ideas. Two teams compete against each other in form of best bugs and session notes. The two teams go through the opposite teams material and conclude who they think should be the winner with a good reason why.

Make a compelling status report

As the last event in the testlab we want to investigate how we can make a compelling status report for our stakeholders. Having many different testers, session notes spread all over, half-finished bug reports and test ideas half-finished… can we create something that is still valuable to someone? I guess this is a common situation at any lab at any company, still we will dig deep into how to go about this.

Part of the line up in the test lab is:

•  Ben Kelly

References

[1] Ultimate Frisbee Rules –  http://www.cs.rochester.edu/u/ferguson/ultimate/ultimate-simple.html

The Tester’s Pocketbook by Paul Gerrard is not a great book, but it is very good.
It covers fundamentals of software testing, and contains a ton of good ideas that will help you in your testing effort.
I also like it because it is one of few books on testing theory that focus on the human element of software, and its creation.

However, I disagree with some things, and will focus on the test design part.
These are Gerrard’s test design axioms with my verdict:

Test Model – Test Design is based on models
Yes, use many, and also look outside them. Build your own quality characteristics model.

Test Basis – Testers need sources of knowledge to select things to test
Yes, use multiple information sources to understand what is important, the list is bigger though (Purpose, Capabilities, Failure Modes, Quality Characteristics, Usage Scenarios, Creative Ideas, Models, Data, Surroundings, White-box, Public Collections, Internal Collections, Business Objectives, Information Objectives, Product Image, Product Fears, Project Risks, Rumors, Product History, Project Background, Test Artifacts, Debt, Business Knowledge, Field Information, Users, Conversations, Actual Software, Technologies, Standards, References, Competitors, Tools, Context Analysis, Legal Aspects, Many Deliverables, Searching, You.)

Oracle – Testers need sources of knowledge to evaluate actual outcomes or behaviors
Very useful, but not necessary; we can sometimes suspend judgment and communicate noteworthy information.

Coverage – Testing needs a test coverage model or models
Not needed, unless as a tool to get more test ideas, or a way to report what has been tested. Gerrard’s question “how many tests remain?” is not good. A better one is “how much test time do you guess remain?”

Prioritisation – Testing needs a mechanism for ordering tests by value
Don’t waste too much time on this; when necessary use a fast, frugal test triage.
Page 38 states “we must invite stakeholders to take an utilitarian view.” This is not true. We could equally well use a value-based system, e.g. “bought software should not crash”, or just go by feelings about what is right.

Fallibility – Our sources of knowledge are fallible and incomplete
True, but text gets a bit too negative towards the human mind. An engaged mind can make mistakes, but also discover what is important. We can separate right from wrong, we can handle the unknown, we can make up for mistakes done.

So which axiom is missing?
Sampling & Serendipity – testing is inevitably a sampling activity, where serendipity is to our help.

There is also an overall problem when Gerrard states that these axioms are needed to do testing. It should be: needed to do good testing.
Nonetheless, one of the better testing books out there!

If you want to seriously improve your bug reporting skills, read up, or take, the BBST Bug Advocacy course.
If you want to start by improving bug report title/subject/summary; read Lessons Learned in Software Testing, no, 83, or this blog post.

Many people will only read the title, so it is important to make it possible to
* understand how the problem appears
* understand limitations and dependencies
* understand the consequences of the bug

Some people will make their first decision on fix/don’t fix, based solely on the title.
(And for those that look carefully at the report, the title will guide their thinking.)
The goal is to make it possible to understand the bug, and how important it is, just by reading the title.

A few tips

* As short as possible, but no shorter than that.
* Start the sentence with the most important, to capture the reader’s interest.
* Don’t overdo “externalization” to capture interest, rather describe the dry facts, and let the readers draw conclusions
* If it’s difficult, try many times, or write the title after everything else is done (you might find the right words on the way.)
* Include a brief description of what happens.
* Include where the problem happens.
* Describe observations rather than (presumed) facts.
* Use a fair description, don’t exaggerate or understate the consequence of the problem

I also have a tiny, controversial one; start the title with lower case for the first word.
Most testers think they are writing a sentence in a story, and start with upper case (and end with a redundant full stop.) The problem I see with this is that you lose the ability to use upper case for names and terminology. “Exit” refers to an element in the software, “exit” refers to the action, which can be done in several ways. This is nitpicking, yeah, but it’s what I think.

You can’t include everything in the title, so use what’s most important.
The best way to learn this comes as no surprise: practice a lot.

While sifting and sorting the ISTQB Glossary 2.1 I finally found a couple of terms which definitions were both correct and useful:

1. deliverable – Any (work) product that must be delivered to someone other than the (work) product’s author.
Good, because it puts focus on the fact that you are creating the deliverable so it can be useful for someone else.

2. user-based quality – A view of quality, wherein quality is the capacity to satisfy needs, wants and desires of the user(s). A product or service that does not fulfill user needs is unlikely to find any users. This is a context dependent, contingent approach to quality since different business characteristics require different qualities of a product.
This is one of five ISTQB definitions of quality that are worth knowing about. This one is also correctly described.

3. walkthrough – A step-by-step presentation by the author of a document in order to gather information and to establish a common understanding of its content.
A sometimes useful practice, and a definition including the magic “common understanding”.

So if someone says that all ISTQB definitions are wrong, I can confidently say I disagree.

Download!

It is often stated, with right, that you should use many, different information sources in order to come up with good test ideas.
Rob Sabourin uses 10 categories, HICCUPPS(F) can be used not only as oracles, and Cem Kaner has many examples in various presentations and tutorials.

We decided to make our own list with sources we find useful, and describe them so they can be useful for others.
We finally ended up with 37 Sources for Test Ideas; have a look and see if they can inspire your quest for what is important.

Comments, additions, experiences are more than welcome!

/Rikard, Martin and Henrik

Generic test report:

 DateTime
 ID
 Author
 Time for writing report
 Time maintaining scripts for generating report
 Application Under Test
 Areas tested

For each area:
 Blockers
 Testers
 Tester mood
 Tests planned
 Tests executed
 Tests passed
 Tests failed
 Tests remaining (untested+fail)
 Bugs found, per priority
 Old bugs status (priority, severity, tester, assignee, days active)
 Bug resolution aggregation

Coverage (overall, or for each area)
 Requirements coverage
 Specification coverage
 Code coverage (statement/decision/condition)
 Data coverage
 Models' coverage
 Model of models coverage
 Serendipity
 For each quality characteristic:
 Importance
 Measurement scale
 Points
 Violations
 Gut feeling

Enhancements
 Noteworthy information
 Other artifacts
 Obstacles
 New risks
 Old risk status
 Learnings
 Test plan change requests

Quality assessment
 Information objectives fulfillment
 Go/No-go recommendation
 upcoming activities

This is of course practically useless.
You should report what is important.

A year has passed since we released version 1.0 of our quality model without metrics.
It is time for a new version, with additions and corrections we have learned over the year (and a Swedish translation!)

1.1 English
1.1 Swedish
1.0 English

Feedback is always welcome!

/Rikard, Henrik, Martin

There are some discussions about current certification schemes, but there is not so much attacks and defense of the actual content.
This is from ISTQB Glossary 2.1:

black box test design technique: Procedure to derive and/or select test cases based on an analysis of the specification, either functional or non-functional, of a component or system without reference to its internal structure.

A first start of the narrowness I dislike is “test cases”.
Why must test design have a set of instructions and expected results?
I think test design can have many forms: detailed, visual, one-liners, tables, un-documented, charters, and it is not good to steer testers towards a limiting format, that in my opinion seldom is appropriate (because it stifles serendipity, promotes confirmation bias, is cumbersome to review, and time-consuming to write and maintain.)

“Analysis” might be the most under-estimated and un-elaborated areas in software testing.
In ISTQB foundation its allotted time is about 2 minutes, and I haven’t seen any interesting on this in the Advanced or Expert syllabi either.

Maybe it is because the test basis only consists of “specification”. I know it happens that tests only stem from specifications, but I can’t understand why.
Don’t we want to find out how the system really behaves?
Do we genuinely believe that the writers captured everything that might be important?
Are we consciously neglecting everything we learn throughout the development project?
Requirements are a good start, but there are a lot more to look at.

Have they written “derive and/or select” to make sure that no creativity and new ideas appear in test design?

That the definition reads “the specification” is a symptom of the un-holistic world view that each function/feature should be tested in isolation.

At least there is mention of “non-functional”, but I don’t want to detail my critique on their view on this (I think it should be done together with other testing, that it doesn’t have to be done by experts, that it is OK that it isn’t measurable in quantitative format, that testers should have a broader view and knowledge.)

I haven’t taken the ISTQB training, with the right teacher it might be great, especially for newcomers that want a glimpse on many aspects of what software testing is about.
But it is a pity that the content is so meek, bleak, weak.

I believe there is too much focus on test coverage , there is even an axiom about the need of it.
My reason is that no coverage model captures what is important.
Cem Kaner lists 101 possible coverage models (Software Negligence and Testing Coverage), and none of them are super-good to me (my favorite is an expansion of no. 89: Potential Usage, which is impossible to measure.)
A dangerous example is coverage by amount of planned tests performed, which easily gives too little exploration, and less ambitious testing efforts.

Test coverage is about planning, precision, measuring and control; which isn’t the best match for things that can be used in a variety of ways, with different data and environment, and different needs.
Sure you can make use of them, but if you rely too much on them, you will have problems in an industry of uncertainty like software development.

The over-emphasis can be shown in the following ISTQB quote:
“Experience-based tests utilize testers’ skill and intuition, along with their experience with similar applications or technologies. These tests are effective at finding defects, but not as appropriate as other techniques to achieve specific test coverage levels or producing reusable test procedures.”
(implying that you can’t really rely on these methods that merely) find defects (and important information.)

I understand that coverage models can give confidence to the decision makers, but how often are these used in reality?
Aren’t release decisions rather made based on how you feel about the facts you are presented with; and it is specific bugs that can stop a release, and external factors that push a release?

If so, isn’t focus on coverage model sort of wasted?
And if it brings a slower testing with less result, it is something to try to get rid of?

As an alternative, I present my 95% Table:

The measurement used is anything you want it to be, and of course practically unusable.

- SO HOW ARE WE GONNA REPORT STATUS? I hear shouted.

In a different, and better way.
I’m not sure how, but I want to be close to what’s important, and far away from John von Neumann’s quote:
“There’s no sense in being precise when you don’t even know what you’re talking about.”

With all due respect, this is the announcement of the perhaps most powerful public two-page document in the history of software testing.
It is an extended re-write of James Bach’s Quality Criteria Categories, and has been developed to 12 categories (CRUCSPIC STMP) and 93 sub-categories for software quality characteristics/attributes/factors/dimensions/properties/criteria/aspects.
This list is not objectively true, and not easy to use for measuring (compare with ISO 9126-1), but you can adapt it to your context and be inspired by it when understanding/creating/reviewing software quality related stuff.

The PDF is available here, and goes under the Creative Commons Attribution-NoDerivates license, as all our Publications.
Use the list to generate test ideas, or to discuss what is important; get inspired by the concept of Charisma (SPACE HEADS mnemonic)
Suggestions for improvements to the list are very welcome!

We will continuously blog about some of the categories (and sub-categories) in order to provide more information and examples of usage.

Rikard, Henrik, Martin
the test eye