Generic test report:

 DateTime
 ID
 Author
 Time for writing report
 Time maintaining scripts for generating report
 Application Under Test
 Areas tested

For each area:
 Blockers
 Testers
 Tester mood
 Tests planned
 Tests executed
 Tests passed
 Tests failed
 Tests remaining (untested+fail)
 Bugs found, per priority
 Old bugs status (priority, severity, tester, assignee, days active)
 Bug resolution aggregation

Coverage (overall, or for each area)
 Requirements coverage
 Specification coverage
 Code coverage (statement/decision/condition)
 Data coverage
 Models' coverage
 Model of models coverage
 Serendipity
 For each quality characteristic:
 Importance
 Measurement scale
 Points
 Violations
 Gut feeling

Enhancements
 Noteworthy information
 Other artifacts
 Obstacles
 New risks
 Old risk status
 Learnings
 Test plan change requests

Quality assessment
 Information objectives fulfillment
 Go/No-go recommendation
 upcoming activities

This is of course practically useless.
You should report what is important.

A year has passed since we released version 1.0 of our quality model without metrics.
It is time for a new version, with additions and corrections we have learned over the year (and a Swedish translation!)

1.1 English
1.1 Swedish
1.0 English

Feedback is always welcome!

/Rikard, Henrik, Martin

There are some discussions about current certification schemes, but there is not so much attacks and defense of the actual content.
This is from ISTQB Glossary 2.1:

black box test design technique: Procedure to derive and/or select test cases based on an analysis of the specification, either functional or non-functional, of a component or system without reference to its internal structure.

A first start of the narrowness I dislike is “test cases”.
Why must test design have a set of instructions and expected results?
I think test design can have many forms: detailed, visual, one-liners, tables, un-documented, charters, and it is not good to steer testers towards a limiting format, that in my opinion seldom is appropriate (because it stifles serendipity, promotes confirmation bias, is cumbersome to review, and time-consuming to write and maintain.)

“Analysis” might be the most under-estimated and un-elaborated areas in software testing.
In ISTQB foundation its allotted time is about 2 minutes, and I haven’t seen any interesting on this in the Advanced or Expert syllabi either.

Maybe it is because the test basis only consists of “specification”. I know it happens that tests only stem from specifications, but I can’t understand why.
Don’t we want to find out how the system really behaves?
Do we genuinely believe that the writers captured everything that might be important?
Are we consciously neglecting everything we learn throughout the development project?
Requirements are a good start, but there are a lot more to look at.

Have they written “derive and/or select” to make sure that no creativity and new ideas appear in test design?

That the definition reads “the specification” is a symptom of the un-holistic world view that each function/feature should be tested in isolation.

At least there is mention of “non-functional”, but I don’t want to detail my critique on their view on this (I think it should be done together with other testing, that it doesn’t have to be done by experts, that it is OK that it isn’t measurable in quantitative format, that testers should have a broader view and knowledge.)

I haven’t taken the ISTQB training, with the right teacher it might be great, especially for newcomers that want a glimpse on many aspects of what software testing is about.
But it is a pity that the content is so meek, bleak, weak.

I believe there is too much focus on test coverage , there is even an axiom about the need of it.
My reason is that no coverage model captures what is important.
Cem Kaner lists 101 possible coverage models (Software Negligence and Testing Coverage), and none of them are super-good to me (my favorite is an expansion of no. 89: Potential Usage, which is impossible to measure.)
A dangerous example is coverage by amount of planned tests performed, which easily gives too little exploration, and less ambitious testing efforts.

Test coverage is about planning, precision, measuring and control; which isn’t the best match for things that can be used in a variety of ways, with different data and environment, and different needs.
Sure you can make use of them, but if you rely too much on them, you will have problems in an industry of uncertainty like software development.

The over-emphasis can be shown in the following ISTQB quote:
“Experience-based tests utilize testers’ skill and intuition, along with their experience with similar applications or technologies. These tests are effective at finding defects, but not as appropriate as other techniques to achieve specific test coverage levels or producing reusable test procedures.”
(implying that you can’t really rely on these methods that merely) find defects (and important information.)

I understand that coverage models can give confidence to the decision makers, but how often are these used in reality?
Aren’t release decisions rather made based on how you feel about the facts you are presented with; and it is specific bugs that can stop a release, and external factors that push a release?

If so, isn’t focus on coverage model sort of wasted?
And if it brings a slower testing with less result, it is something to try to get rid of?

As an alternative, I present my 95% Table:

The measurement used is anything you want it to be, and of course practically unusable.

- SO HOW ARE WE GONNA REPORT STATUS? I hear shouted.

In a different, and better way.
I’m not sure how, but I want to be close to what’s important, and far away from John von Neumann’s quote:
“There’s no sense in being precise when you don’t even know what you’re talking about.”

With all due respect, this is the announcement of the perhaps most powerful public two-page document in the history of software testing.
It is an extended re-write of James Bach’s Quality Criteria Categories, and has been developed to 12 categories (CRUCSPIC STMP) and 93 sub-categories for software quality characteristics/attributes/factors/dimensions/properties/criteria/aspects.
This list is not objectively true, and not easy to use for measuring (compare with ISO 9126-1), but you can adapt it to your context and be inspired by it when understanding/creating/reviewing software quality related stuff.

The PDF is available here, and goes under the Creative Commons Attribution-NoDerivates license, as all our Publications.
Use the list to generate test ideas, or to discuss what is important; get inspired by the concept of Charisma (SPACE HEADS mnemonic)
Suggestions for improvements to the list are very welcome!

We will continuously blog about some of the categories (and sub-categories) in order to provide more information and examples of usage.

Rikard, Henrik, Martin
the test eye

I saw an interesting interview with Rob Sabourin today http://www.youtube.com/watch?v=HZRXdaN7gkY  (Thanks for the tip, Jon Bach!)
One thing he says in this video is: “… There are a lot of template junkies out there. [Testers are]  filling out templates and not actually testing. That frustrates testers…”

Hey, isn’t this the same thing that happens in strictly scripted testing environments?
When testers are following detailed test scripts they are actually ending up filling out templates. With the same frustration as when filling out templates where you also don’t have to bother use your brain.
The result is the same for scripted testing as with a filled out template. Little mind work has been involved and so the information is very narrow and/or shallow…

It is sometimes said that templates are good because it is prevent someone from doing mistakes and/or making sure that the format is the same all the time. But isn’t a template to a document the same as what a script is to a test? A prescribed recipe on how to complete the task. I.e. it is about management control.

Isn’t it so that you easily can be hit by inattentional blindness as you fill out the template and only worrying about not screwing up the format? As with scripted testing, the focus lies in the format rather in the content. Both in scripted testing and in filling out templates you worry about making sure all things are filled out as expected, not the actual testing or the actual writing of informative text.

And, as with scripted tests, if we could fill out the templates automatically, we wouldn’t need to do them manually.

At many bigger companies where you have several teams working with the same test scope there is often a need to communicate how you intend to test something and what areas you intend to cover. It is common that you have a test plan and perhaps a test strategy, they usually do not go into detail on each feature or area that you intend to test. Instead you create a test specification containing test cases with repro-steps and lots of information so that anyone can execute the test case. The test specification need to be reviewed before you commence with testing. Many times you assign one person to be responsible for creating such a spec. It is not uncommon that you set someone to write on this test specification for several weeks. If you have never seen this situation, then good for you!

The rest of you who also see this as a fairly common situation… here is a different way of working with test specifications (or rather not working with them).

Let us assume that you are a member of a test team that is on one test site out of many in a project. The head project manager asks you, as a test lead, to give an estimation on how much time you need to system test feature X. At the same time the head project manager asks another test lead on how to test feature X, but at a lower test level such as integration or function. Instead of giving an estimation directly, the test leads asks to prepare and plan a bit then come back shortly.

Instead of doing it the regular way, as in both test leads giving an estimate to the project manager without communicating or writing down their thoughts, one of the test leads start writing on a test document, namely a Test Proposal (TP). All ideas, risks, work packages, unresolved issues, todos, estimations, contact persons, etc are listed in this TP. The test lead can show the rest of the team the thoughts that have guided him so far and can assist with the rough estimation. Shortly they can return to the project manager with an estimate and explain roughly the plan outlined in the TP. It is not uncommon that some of these features are not able to be added to the overall project, so the test lead does not spend any more time at this stage. If possible the two test leads have cooperated and written in the same TP, but worked on different perspectives and areas. A lot of information would have been the same for both test leads.

A few weeks later feature X appears on the radar to be included in the release after all. The test leads or those responsible for the feature can now continue preparing. A group of two or three testers, depending on size of feature, form up to further prepare the feature. If the feature is available to be looked at in the release, the group starts do a minor test in order to understand how it worked and getting a feeling what to expect. Then they sit down and brainstorm on the feature further. More content is added to the TP and eventually it is ready to be shown to business analysts and developers responsible for the feature as well as to the other test teams working on the same feature. The TP is quite short because of the one-liner test ideas, thus effective to be communicated and reviewed. After a few rounds between the different parties the list of test ideas, open issues, work packages and estimations etc begin to feel ready (as far as ready can be in testing). The TP is therefore approved and the test teams are considered to be prepared to start testing. The total time spent is probably a few days at the most.

Comparing this method with how test specifications and estimations are done, you will by using this method not loose as much information since you collect your thoughts in the TP. It is otherwise common that the person doing the estimation does not give any information on what was included.

Reviewing and communicating a test specification containing all your test scripts with all your repro-steps is incredibly hard. It is not effective to give feedback on huge documentation, no one usually have that much time to spare. Instead the TP only contains the guiding thoughts and is easy to review. A test idea is usually a one-liner with the essence of the test intended, which is perfect for communication.

The key to this is collaboration and cooperation in the team as well as with other stakeholders. You focus on learning a new feature and the domain around it while at the same time hone on your test design. The power of the TP lies in it being focused on the essentials of the planned testing where you do not go into too much detail instead have the rough scetches and therefore making it easy to communicate. This method works well in combination with SBTM and other styles of exploratory testing.

I have written a small article for The Testing Planet about Testing & Creativity. (update: web article here)
It contains the potato from this blog, but also some new content on “cheating”.
Feel free to add comments here!

I think creativity still is too little recognized as an important aspect of our work.
We should promote it more, both because it is a vital aspect, but also to make the profession more interesting to the next generations.

There are (at least) two upcoming presentations on creativity in software testing:
Rob Lambert – Is excessive structure killing our creativity, blog, video, program.
Rikard Edgren – 5 Steg Till Kreativ Testning, SAST 15 år (a Swedish version 2 of EuroSTAR 2007 presentation Where Testing Creativity Grows)

There are a lot of other content in The Testing Planet paper, my favorite being a compelling story from Scott Barber on the publically available Black Box Software Testing course at testingeducation.org.

One of Cem Kaner’s many classic writings is “What is a Good Test Case?”
It is a very good article, well-spent time for anyone involved in software testing.
But when writing about test ideas, I started to realize that the list of properties for good test cases isn’t perfect, for me.
So it’s time for some criticism of this part of the professor’s article.

His list of attributes for good test cases (for the context “Tests Intended to Expose Defects”) goes something like this:

• Powerful – a test is more powerful if it is more likely to find bugs
• Yield significant results – the issues found are important (to stakeholders)
• Credible – the actions in the test are realistic (not corner cases “noone woud do”)
• Likely – the test reflects how customers actually will be using the product
• Easier to evaluate – ease of telling if the test passed or failed
• Useful for troubleshooting – so it is easy to find out what went wrong during the test
• Informative – regardless of pass/fail status, you get valuable information from the test
• Appropriately complex – if there are many bugs in the software, the test might fail too quickly
• Giving insightful information – the test might not render bugs, but other important information

Test cases can be seen as a broad spectra, from “classic” test cases with exact steps and expected results, to vague, one-liner test ideas (that also could be ongoing, unorthodox or unverifiable.)
So I have to disagree with “Easier to evaluate”, it’s a valuable property in many situations, but not in so many that it should make it on this list. But it’s back on the list if it is named something like “Hints for Evaluation”.

But now to the most interesting part of reviewing: what’s missing on the list?
There are many more things that could be important, a quick search gave:

• Ease of maintenance, Ease of creating variations of it (Shrini Kulkarni)
• Accurate, Economical, Reusable, Tracable, Self-cleaning (Dianne L. Runnels)

These are good things, but not needed on my list (and I’d prefer Fast to execute as name instead of Economical.)
But I have two other things I would like to add

• Easy to understand – to enable reviewing by different types of people (more likely to happen for one-liners)
• Enables serendipity – the test is rich in the sense it has possibilities of finding issues other than the ones the test case is aiming for

The serendipity can either be covered inside the test case, or by allowing variations, or by putting freedom and responsibility on all testers to deviate and look at more things, if deemed worthwile, while executing the test cases.

…and always remember Kaner’s advice that “Test cases can be “good” in a variety of ways. No test case will be good in all of them.”