Status reporting of testing activities is extremely project-dependent. The needs of when and how and what will differ every time. Maybe that’s why there’s so little good writing about status communication; you have to make it up every time.

Templates are out of the question, and I believe examples will mislead you as well.
You’re better off by thinking around these questions (inspired by final section in Lessons Learned)

Why are you reporting?

Are you giving quality assessments, trying to prove why testing is taking place?
Is focus on the product story or the testing story?
What kind of information can change decisions that are based on your report?

What is most important?

The everlasting and most difficult question for all testing efforts.
This is what you should start your report with.

To whom are you reporting?

What kind of information is the audience interested in?
Are developers the only ones reading bug reports?
Do you have a grounded quality model?

Are you adjusting the language for the audience?

Communication includes making sure it is understood.

In which ways are you reporting?

Just one way is rarely enough; use talking, writing, long, short, formal, informal.

Are there certain times when things need to be communicated?

Project milestones and meetings might need your information.
Some things must be communicated continuously and timely; bringing up bad news on the last day is a very bad idea (but sometimes inevitable.)

Are you using status reporting as a means for new test ideas?

I often start writing an important report early, to identify things that needs more testing.
You should start thinking about the reporting before the testing starts.

Are you reporting just because you are supposed to?

Then you should read these questions again…

If status reporting is difficult, there might be something wrong with your testing.

- I saw you tested “42″. How come you didn’t try “43″?
- That’s obvious. It would be the same. No need to test something that would give the same result.
- OK, so I guess you are familiar with equivalence partitioning?
- Beg your pardon?

It is quite often said that testers don’t know the basic test design techniques.
This is both true and false.
“All” testers know how to use basic equivalence partitioning, it is common sense: you put things in groups, so you don’t have to test similar things that would yield the same information.
But they don’t know the name, and they don’t know the theory behind partitioning many types of groups.
But if they are testing all important partitions, is it a problem?

Might be, but probably not. I think a tester with knowledge about the product and its usage will make a common sense selection of the partitions that are needed to be tested for that particular situation.
For cases where you need to be very thorough this will include inputs, outputs, combinations; and extensions to examples like these:

Numericals: normal, inside/outside boundaries, min/max data type, different data types, decimals, precision
Strings: A-Z, ÅÄÖ, Unicode, empty, many, extremely many, control chars, leading/trailing spaces, things that mean other things (*, SQL, links, special values)

- You know you can’t be called a real tester if you don’t know how to do equivalence partitioning.
- Be so it. I just test what is important in my project.

Storytelling has been rising for quite some years and it will soon boom for software testing.
The reason is simple: people like stories.
And if it is used as status reporting instead of lame numbers, it is a step in the right direction, to say the least.
But when testing this idea theoretically, I find some fears:
* they will take long time to tell
* they will be too entertaining, less informative

And when comparing with my experiences, I find success stories when remembering conversations where I fast communicated the essence.

We already have a core tester skill in writing effective bug titles.
We should learn how to do this for whole test efforts, we should always have an executive summary up our sleeve.
We shouldn’t tell stories, it should be more like a librarian’s summary.

To accomplish this we need a lot of training, but I doubt it will be enough.
We also need more words.
They don’t have to be unanimously defined, and they can be reconstructed at each company, as long as they are useful.
Do a daily exercise of explaining the state of the product in thirty seconds, and try out your own terminology.

We need more words to effectively communicate the essence of our findings.

James Bach and Michael Bolton has a classic collection of consistency oracles; HICCUPPS(F): History, Image, Comparable Products, Claims, User Expectations, Product, Purpose, Standards and Statutes, Familiarity

It is a very good collection; not only helping you find out if something is a problem or not, but also the other way round: serving as testing inspiration.
At EuroSTAR 2009 tutorial Exploratory Testing Masterclass, Michael said that anyone coming up with additions would become famous.
As a tester this was of course an interesting challenge!
So over time, I now and then have thought of additions, but they haven’t been solid.
For example; V as in Violation of Quality Characteristics, could be seen as an inverted Familiar Problems or User Expectations.
You, as in the tester’s subjectivity can also be part of F and U.

But finally I found the important oracle missing from their list.
C as in Conversations.
You talk to a developer or other stakeholder, and collaborate to reach a decision.
Yes, during the discussion you might use HICCUPPS(F) oracles, but not necessarily. And it might not be the whole story, and not your oracle.
It might not be a 100% consistency heuristic, but is important enough to not leave out.
C as in Consistent with Conclusions from Collaborative Communication.

Outside the list we have the important No Oracle, where judgment is suspended, and noteworthy information is communicated, without deciding if it’s a problem or not.
But that doesn’t make for a nice soccer-sounding mnemonic, and is better left as a side-note to HICCUPPS F.C. heuristic.

(When double-checking, I noticed that they recently have added another one in the RST slides: Explainability – The system is consistent with my ability to explain it.
For now, I’ll put that one as a sub-part of Conversations; in dialogue, we weren’t able to explain the behaviour.)

History - the product should be consistent with previous versions
Image - consistency with the looks and behavior expected of the product
Comparable Products - consistency with competitors, references or other solutions to similar problems
Claims - consistent with written or oral statements about the product
User Expectations - consistent with what a reasonable user would expect
Product - consistent within itself
Purpose - consistent with the reasons for making/using the product
Standards and Statutes – consistent with standards or other authorities
Familiarity - inconsistent with problems seen before
Conversations - consistent with conclusions from dialogue

During my first paternity leave I learned sourdough baking. During the second I couldn’t help writing an ambitious paper, or a small book, about people-oriented test design, about things beyond test design techniques, close to the exploratory testing tradition.

It can be downloaded here.

It contains collections of knowledge, and generalizations of my ten years of testing the same product suite. I think it can be useful for ambitious testers that want to find any problems that might be important.

It probably is too much, theoretical, irrelevant or condense for many of you, but if you want to give it a shot I recommend the following:

* Download The Little Black Book on Test Design
* Print as double-sided A5 Booklet
* Find a quiet, comfortable place
* Read and relate to your test reality

Comments are welcome, especially additions to the collection of one hundred and three test design heuristics.

How do you find out what is important, in your specific situation?

I think it is the essential problem for all activities with complexity.
I think it is impossible to do really good software testing without the ability to dismiss things as not important, and dig deeper for matters that are important (I believe it’s the same for cooking a meal, or raising a child.)
I don’t think you can sample appropriately unless you know a lot more than the requirements.
I think we need better understanding and models for this, if we are to persuade those that want to govern by number; they aren’t hitting the crucial things.

It doesn’t seem like pointing to intuition and subjectivity is enough, but I haven’t reached further than this:

Either you know what is important (when you see it), or you don’t.
So the more knowledge you have about things that matter, the better suited are you to find important quality-related information; some as planned, some by serendipity.
So the key is to learn a lot, from a variety of sources.

I obviously need help on this one…

I know of some testers who are pointed out by others to be Bug Magnets; people recognized for their ability to somehow draw bugs to them. Bug Magnets can be found in many workplaces and I bet that you know of someone that falls under this description. I have been appointed a Bug Magnet by some and it have made me thinking on what this phenomena boils down to.
Is it luck? Is it faith? Is it an ability that some are born with and some aren’t? Can you learn this ability? Can you improve it?
I have wondered about this for some time.

However, this summer I had a revelation when I watched an episode of the marvelous TV series “Homicide: Life on the Streets”:

Homicide: Life on the streets
Season 1 Episode 9 “Night of the dead living”
Det. Frank Pembleton and Det. Tim Bayliss

Bayliss [sits pensively. To Frank]: What are you looking at?

Pembleton is playing cat’s cradle.

Bayliss: You have something that you wanna say to me?

Pembleton: Adena Watson. So many unanswered questions.

Bayliss: And you’re saying that I’m not asking them.

Pembleton: I’m saying that you’re not answering them. [He peers at Tim through the cat's cradle.]

Bayliss [sighs]: What questions aren’t I answering, huh?

Pembleton [gets up and flips through a notebook, draws a picture]: Okay, these sixteen row houses on the north side of Kirk Avenue. Adena’s body was found outside the kitchen door in the red yard at 718 Kirk. Now the killer could have dropped her anywhere. Why not the common alley? Why not the yards at either end of the block? These three row houses are empty. One, two, three. The killer would have stood much less of a chance of being seen if he’d dumped her body in any one of these yards. Why would he risk bringing a little girl’s body inside a closed fence of an occupied house?
Maybe he wanted her body to be found immediately. Maybe he wanted to cast suspicion on the people in 718. Maybe he had some … perverse sense of remorse, some impulse to leave her body inside an enclosed yard to protect her from stray dogs.

Bayliss: These are *exactly* [taps the notebook] the questions that I have been trying to answer.

Pembleton: Well, you can try, but you never will.

Bayliss: Why?

Pembleton: You don’t think like a criminal. You don’t have a criminal’s mind.

Frank walks away. Tim grimaces in disbelief.

Bang! Suddenly it struck me. “You don’t think like a criminal. You don’t have a criminal’s mind”.
In order for a police to think of possible outcomes of a crime they have to be able to think as criminals, and put themselves in a criminal’s way of thinking.
If you don’t do this, you are trying to understand and explain the crime based on your logic.
Similarly, in order to “attract” bugs you have to wanna see problems; you have to identify problems that might bug several kinds of stakeholders; you have to put all your knowledge about the project in to consideration; and you have to be able to see the problems that matter. You are not trying to understand and explain the system by using your own logic, instead you are using several input sources to do this: logic, subjective thoughts, people’s skill levels, complexity of system, technology, etc.

So let me present my take on a definition of “Bug Magnet”:

Being a Bug Magnet = Able to foresee possible problems (or able to spot opportunities for things to go wrong), in context.

The most important thing here is the last two words “in context”. That is, even if you have all the bug taxonomies and oracles in the world to support you, you have to be able to understand what matters in this project.
Knowledge about “all common problems in .NET applications” can help you sometimes. Knowledge about “all common problems in .NET applications that developer X often produce”, is however much more useful.
Understanding what bugs our users is more helpful than knowing about what bugs users in general.
Knowing about problems with focus in Windows applications is one thing, finding these problems during testing is to be able to spot opportunities when they are presented to you in your context (see http://thetesteye.com/blog/2010/11/windows-focus/ ).

Now, back to the questions in the beginning of the post.

• Is it luck?
  – No, even if luck sometimes help. For others it is more of welcoming serendipity.
• Is it faith?
  – I don’t believe in faith.
• Is it an ability that some are born with and some aren’t?
  – Maybe. Some people might have a more developed talent, but I think that most people have this talent. Some are born with variations of narcissistic personality disorders and might have difficulties with this.
• Can you learn this ability?
  – I believe so.
• Can you improve it?
  – Yes. However, you might need to consider one or several dimensions to improve: Empathy, reasoning, attention for detail and seeing the whole, recognizing patterns of your own and other peoples mistakes, subjectivity,  general systems theory, context-driven testing, and more.
  Notice that these dimensions are not technical but rather comes from social sciences.

In testing text books you can read that compatibility testing should be performed after the functionality testing and bug fixing is completed. I guess the reason is that you don’t want to mix these categories, but hey, what a waste of resources.
My suggestion is to perform the compatibility testing at the same time as you are doing your other testing; when problems arise, trust that you will deal with them.
In my classification, compatibility testing involves hardware, operating system, application, configuration, backward/forward compatibility, sustainability and standards conformance.
Here follows some lightweight methods to tackle these areas.

Basic Configuration Matrix

Basic Configuration Matrix is a short list of platform configurations that will spot most of the platform bugs that could exist in your currently supported configuration matrix.
The simplest example is to use one configuration with the oldest supported operating system, oldest browser etc; and one configuration with the newest of all related software. A more advanced example could use several configurations that use different languages, Application Servers, authentication methods et.al.
Often it will take quite some time to run most tests on BCM; so alternate between the configurations while testing your product. Do variations on configurations when appropriate.

Error-prone Machine

Another trick is to setup machines so they have a high chance of stumbling on compatibility issues. You can vary this on your BCM, your personal machine or whatever is suitable. The idea is to get some compatibility testing almost for free.
Examples on Windows include: run as Restricted User, use Large DPI setting, use German Regional Settings, install support for all of the worlds characters, non-English language in Internet Browsers, move system and user Temp folder, activate IE ‘display a notification about every script error’, move Task Bar to the left side of the screen, Windows Classic Theme & Color Scheme, use User Access Control, use an HTTP proxy, use Data Execution Prevention, install new versions as they come, e.g. latest hotfixes, MDAC etc, never install software on default location, run with 2 screens, on a 64-bit system, use different browsers, turn off virtual memory swapping, , install Google/Yahoo toolbar, run Energy Save Program, pull out the network cord every time you leave the computer; and put it back in when you return, turn on the sound!

Technology Knowledge

If you know a lot about the environment the software operates in, you know which things will happen in reality, which settings that usually are altered, and how it is commonly operated.
The lightweight method is to use this knowledge and make sure you test the most important things.

Letting Others Do the Testing

Many compatibility problems happen on basic usage, which indicates that you can let others do a big part of the compatibility testing: developers can use different machines and graphics cards, Beta testing can be done in customers’ production-like environment. If the product is free of charge, you might even get away with addressing problems after your users encounter them (but make sure you have an easy and compelling way for them to do this reporting.)
Crowd-testing could be a way, but so far the payment models from the testers’ perspective are not ethically defensible, to me.

Reference Environments

To quickly investigate if you are experiencing a compatibility issue, it is handy to have reference environments available. It could be someone else’s, a virtual machine, a quickly cloned image, your own machine etc.
Personally I prefer having a physical machine that is running similar things, but on a different OS, different language and/or earlier version. The last years I have had three machines and three monitors, and by switching, I get a lot of compatibility testing done at the same time as testing new features. When I check things on an older version, I can save documents and use them for next tests.

Backward/Forward Compatibility

Backward compatibility is easiest done if you can use real customers most complicated files/data/documents. Use these as you test any functionality (Background Complexity Heuristic).
Occasionally communicate between different versions.
Forward compatibility should be designed in the architecture, as a tester you can point this out.

Sustainability

Have conversations around the question: Is the product compatible with the environment? Have we considered energy efficiency, switch-offs, power-saving modes, support work from home and the likes?

Standards

A lightweight method for standards conformance is to identify which ones are applicable, and ask the experts if they understand it, and successfully have managed to adapt it to the new context.
Let’s finish with a non-lightweight method: you can become the standards expert.

I used to be a bit skeptic towards the word “heuristic”. It seemed needlessly advanced, and things are possible to explain in other words.
But when I read Gigerenzer’s Gut Feelings about how to catch a flying ball, it all came together.
For software testing, which can’t be complete, is dependent on many factors, with a product to be used with different needs and feelings; techniques are not appropriate. It is about skill, and human skill is very good to describe with a variety of heuristics.

When blogging about some heuristics I think are un-published and worth knowing about, I’ll try to do two at a time; more bang for the bucks.
With English as second language it is difficult to give them good names, so feel free to suggest better names! (and content…)
And remember, heuristics are not rules; they are more like rules of thumb, that might be useful, in specific situations.

No Flourishes Heuristic

At many times you can design and execute straightforward tests, without garnish, fancy tools, and incomprehensible details.
Try this when you have the chance!
See if perceived performance, manually clocked, can give good information.
Use common options instead of combinatorial.
Look at the GUI instead of automating tests.
Try to do something valuable instead of covering all paths in last years use case.
Write your basic test strategies in plain English, so everyone can review them.
Use what you have, and look for what’s important.

New Connections Heuristic

How do you “discover” what is important?
I think it often is about combining different knowledge in new ways.
So you need to know a lot about the product and its context, and look for connections between the wide diversity of knowledge you have.
When reading, talking, thinking, you sometimes instantly think “but this could have big impact if one is trying to do that!”
Or during test execution, when you suddenly get an impulse that you have to add some other things to the stew.
This might seem unstructured, and dependant on chance, but that is OK. If software testing is a sampling problem, we need different ways of discovering what is important.

I have completed a trilogy on the theme The Eye of a Skilled Software Tester

edition 1: Lightning Talk, Danish Alliance, EuroSTAR 2010

edition 2: Article, The Testing Planet, March 2011 – Issue 4

edition 3: Presentation, Scandinavian Developer Conference, april 2011

Some things have changed over time; in the first two I didn’t focus on the most important “look at many places”, besides specifications we need to know about business usage, technology, environments, taxonomies, problem history, standards, test analysis heuristics, quality characteristics, and more…

I also did the necessary switch from errors/bugs to problems; because it is broader, and better pinpointing the psychological paradox: “want to see problems“, things that make Done further away.

While at it, I uploaded a presentation from SAST Öresund yesterday.
77 Test Idea Triggers, a presentation I’d be happy to give again!