thoughts from the test eye

I spent two days in Malmö attending a developer conference with a fantastic test track (put together by Sigge Birgisson.)

I did a presentation on Curing Our Binary Disease (slides, abstract), which was much better received than I hoped for (I thought it was a binary love/hate talk)
Good questions and talk about being inside the potato, how do you know where you are?

Pradeep and Rikard in front of Øredev projector

Pradeep Soundararajan talked convincingly about caring for the users, e.g. by using Twitter as a source for test ideas.
He also said the context-driven community has moved from Pass/Fail to “Is there a problem here?” (still a binary question though…)

Shmuel Gershon shared an experience report of a 100% exploratory testing project, with qualitative reporting and everything (“it’s called quality report, not quantity report”.)

Gojko Adzic talked about testers needing to adapt to how development is done now, and put most of their time ensuring others write good automated tests.
“There’s so much mistrust in the processes.”

Henrik Andersson wants, with right, a diversified testing team, and therefore only hires context-driven testers 😉
Just joking, he said people can read the same excellent books, as long as they think critically and learn other things.

Selena Delesie explained how to focus the testing effort on customer needs; “testers are information radiators”.

Zeger van Hese made the presentation that I enjoyed the most. A thoughtful walkthrough similarities between the fine arts and testing.
“The Hungry Eye – thoughtfully looking at software”

Janet Gregory highlighted five enlightened areas since Agile Testing was released: feature acceptance, collaborative automation, large organizations, distributed teams, continuous learning.

Outside the venue I had many interesting conversations, there were testing games, food and drinks, Black Viper and even imitations, and all in all a very nice experience.
Thumbs up for a testing session with Pradeep and Shmuel that showed (the need for) different note taking styles.

The presentations will be made available online at www.oredev.org/

A year has passed since we released version 1.0 of our quality model without metrics.
It is time for a new version, with additions and corrections we have learned over the year (and a Swedish translation!)

1.1 English
1.1 Swedish
1.0 English

Feedback is always welcome!

/Rikard, Henrik, Martin

As an exercise, try these generic test ideas on any product, for instance a recent upload to sourceforge.net. Then come up with a better test idea, and write as a comment. Also suggest one to remove; I have decided they must be seventeen (a Tranströmer homage)

* Try to do what it is supposed to do
* Verify that Help matches functionality
* Let it run for quite some time, and look at resource utilization
* Play around with core functionality for five minutes
* Ask someone else for their first impression
* Perform any simple scenario as fast as possible
* Use keyboard-only for a while
* Have a look at all installed files
* Get rid of all traces of software (Uninstall)
* Run on a radically different platform, preferably an Error-Prone Machine
* Use credibly dirty data
* Provoke a bunch of failures
* Be a user that want to destroy for others
* Compare with common behavior for the environment
* Review About dialog information
* Is it at least as good as a comparable product?
* Do something valuable for you

Investigate your feelings during these tests; keep usability, charisma, security, performance and testability in the back of your head.
End by describing the state of the product in less than a minute.

These are test ideas you can try on any product, and they will give you interesting information.
However, spending the same amount of time on other tests, might be even better…
Really good test ideas are specific for a product, capturing their essence; challenging a main purpose.

Storytelling has been rising for quite some years and it will soon boom for software testing.
The reason is simple: people like stories.
And if it is used as status reporting instead of lame numbers, it is a step in the right direction, to say the least.
But when testing this idea theoretically, I find some fears:
* they will take long time to tell
* they will be too entertaining, less informative

And when comparing with my experiences, I find success stories when remembering conversations where I fast communicated the essence.

We already have a core tester skill in writing effective bug titles.
We should learn how to do this for whole test efforts, we should always have an executive summary up our sleeve.
We shouldn’t tell stories, it should be more like a librarian’s summary.

To accomplish this we need a lot of training, but I doubt it will be enough.
We also need more words.
They don’t have to be unanimously defined, and they can be reconstructed at each company, as long as they are useful.
Do a daily exercise of explaining the state of the product in thirty seconds, and try out your own terminology.

We need more words to effectively communicate the essence of our findings.

As a tester you find lots of things that bugs you when exploring a system. In some cases these issues only nudge you slightly at first, but after passing over the same issue many times it really starts to drive you crazy. This, at first small issue, has now become something that affect you more than you initially thought.

An example:
A feature in my Samsung mobile is annoying. I have a bluetooth headset. When I switch this on and enter the bluetooth settings on the mobile I cannot find the device the first time I enter or scan for it. I must go back then re-enter to find it. I do this every time I connect the bluetooth, which is a few times every day. After having the phone for a month it starts to irritate you. It might be my headset that is buggy or might be the way bluetooth is done on the Samsung. Either way, my experience of my Samsung is damaged and especially this issue is something that nudges me continuously.

When we are testing and report an issue like this, it is usually set to minor severity and as times goes by we want it to be of higher severity. As I see it, it is your obligation to update the bug with your new feelings and reactions. Because our first reaction to something might not be what we feel about the same subject further on. Our initial estimation on severity and priority of bugs should therefore be continuously revised and rethought, if time permits that is.

For many testers you will start to ignore the issue and not see it anymore, thus becoming biased [1]. There is also a big chance that you will suffer from the effect of the Broken Window Theory [2] by accepting similar issues to pass by. It is my belief that we very often give a faulty set of priority and severity when we do not know the system well, but the more we learn and understand the better chance we have to know how important and serious the bugs are.

References

[1] http://www.workroom-productions.com/papers/The%20Irrational%20Tester%20v1-06.pdf

[2] http://www.theatlantic.com/magazine/archive/1982/03/broken-windows/4465/

James Bach and Michael Bolton has a classic collection of consistency oracles; HICCUPPS(F): History, Image, Comparable Products, Claims, User Expectations, Product, Purpose, Standards and Statutes, Familiarity

It is a very good collection; not only helping you find out if something is a problem or not, but also the other way round: serving as testing inspiration.
At EuroSTAR 2009 tutorial Exploratory Testing Masterclass, Michael said that anyone coming up with additions would become famous.
As a tester this was of course an interesting challenge!
So over time, I now and then have thought of additions, but they haven’t been solid.
For example; V as in Violation of Quality Characteristics, could be seen as an inverted Familiar Problems or User Expectations.
You, as in the tester’s subjectivity can also be part of F and U.

But finally I found the important oracle missing from their list.
C as in Conversations.
You talk to a developer or other stakeholder, and collaborate to reach a decision.
Yes, during the discussion you might use HICCUPPS(F) oracles, but not necessarily. And it might not be the whole story, and not your oracle.
It might not be a 100% consistency heuristic, but is important enough to not leave out.
C as in Consistent with Conclusions from Collaborative Communication.

Outside the list we have the important No Oracle, where judgment is suspended, and noteworthy information is communicated, without deciding if it’s a problem or not.
But that doesn’t make for a nice soccer-sounding mnemonic, and is better left as a side-note to HICCUPPS F.C. heuristic.

(When double-checking, I noticed that they recently have added another one in the RST slides: Explainability – The system is consistent with my ability to explain it.
For now, I’ll put that one as a sub-part of Conversations; in dialogue, we weren’t able to explain the behaviour.)

History – the product should be consistent with previous versions
Image – consistency with the looks and behavior expected of the product
Comparable Products – consistency with competitors, references or other solutions to similar problems
Claims – consistent with written or oral statements about the product
User Expectations – consistent with what a reasonable user would expect
Product – consistent within itself
Purpose – consistent with the reasons for making/using the product
Standards and Statutes – consistent with standards or other authorities
Familiarity – inconsistent with problems seen before
Conversations – consistent with conclusions from dialogue

This Wednesday I held a presentation at DevCon 2011 entitled Exploratory Test Design (slides)
I like this terminology a lot, because it encompasses the two things I want to see in software testing
* looking at a lot more information sources than requirements
* vary execution and look for many things

It was also a good presentation experience, because I could cherry pick my favorite heuristics, and talk freely about the ones I care most about, right now.
The audience was mainly programmers, and they are equally interested in exploratory testing and software potatoes!

But the reason I write this post is something Joakim Holm said at his talk, that should be told more often:
if you do a lot of testing, development goes faster

When testing provides continuous feedback, developers understand what is good, what is unreliable, and also what is important.
Continuous corrections towards the goal speed up the time to completion.
Testing isn’t a role with its own goal, we are there to help; and I think the most important help is speed, and the second one is quality (acceptable quality can be met in many ways, with testing it is faster.)

Someone might object and say “on the contrary, with all their phases and complaints, testing makes things much slower!” so I will rephrase:
good testing makes software projects go faster

During my first paternity leave I learned sourdough baking. During the second I couldn’t help writing an ambitious paper, or a small book, about people-oriented test design, about things beyond test design techniques, close to the exploratory testing tradition.

It can be downloaded here.

It contains collections of knowledge, and generalizations of my ten years of testing the same product suite. I think it can be useful for ambitious testers that want to find any problems that might be important.

It probably is too much, theoretical, irrelevant or condense for many of you, but if you want to give it a shot I recommend the following:

* Download The Little Black Book on Test Design
* Print as double-sided A5 Booklet
* Find a quiet, comfortable place
* Read and relate to your test reality

Comments are welcome, especially additions to the collection of one hundred and three test design heuristics.

How do you find out what is important, in your specific situation?

I think it is the essential problem for all activities with complexity.
I think it is impossible to do really good software testing without the ability to dismiss things as not important, and dig deeper for matters that are important (I believe it’s the same for cooking a meal, or raising a child.)
I don’t think you can sample appropriately unless you know a lot more than the requirements.
I think we need better understanding and models for this, if we are to persuade those that want to govern by number; they aren’t hitting the crucial things.

It doesn’t seem like pointing to intuition and subjectivity is enough, but I haven’t reached further than this:

Either you know what is important (when you see it), or you don’t.
So the more knowledge you have about things that matter, the better suited are you to find important quality-related information; some as planned, some by serendipity.
So the key is to learn a lot, from a variety of sources.

I obviously need help on this one…

I know of some testers who are pointed out by others to be Bug Magnets; people recognized for their ability to somehow draw bugs to them. Bug Magnets can be found in many workplaces and I bet that you know of someone that falls under this description. I have been appointed a Bug Magnet by some and it have made me thinking on what this phenomena boils down to.
Is it luck? Is it faith? Is it an ability that some are born with and some aren’t? Can you learn this ability? Can you improve it?
I have wondered about this for some time.

However, this summer I had a revelation when I watched an episode of the marvelous TV series “Homicide: Life on the Streets”:

Homicide: Life on the streets
Season 1 Episode 9 “Night of the dead living”
Det. Frank Pembleton and Det. Tim Bayliss

Bayliss [sits pensively. To Frank]: What are you looking at?

Pembleton is playing cat’s cradle.

Bayliss: You have something that you wanna say to me?

Pembleton: Adena Watson. So many unanswered questions.

Bayliss: And you’re saying that I’m not asking them.

Pembleton: I’m saying that you’re not answering them. [He peers at Tim through the cat’s cradle.]

Bayliss [sighs]: What questions aren’t I answering, huh?

Pembleton [gets up and flips through a notebook, draws a picture]: Okay, these sixteen row houses on the north side of Kirk Avenue. Adena’s body was found outside the kitchen door in the red yard at 718 Kirk. Now the killer could have dropped her anywhere. Why not the common alley? Why not the yards at either end of the block? These three row houses are empty. One, two, three. The killer would have stood much less of a chance of being seen if he’d dumped her body in any one of these yards. Why would he risk bringing a little girl’s body inside a closed fence of an occupied house?
Maybe he wanted her body to be found immediately. Maybe he wanted to cast suspicion on the people in 718. Maybe he had some … perverse sense of remorse, some impulse to leave her body inside an enclosed yard to protect her from stray dogs.

Bayliss: These are *exactly* [taps the notebook] the questions that I have been trying to answer.

Pembleton: Well, you can try, but you never will.

Bayliss: Why?

Pembleton: You don’t think like a criminal. You don’t have a criminal’s mind.

Frank walks away. Tim grimaces in disbelief.

Bang! Suddenly it struck me. “You don’t think like a criminal. You don’t have a criminal’s mind”.
In order for a police to think of possible outcomes of a crime they have to be able to think as criminals, and put themselves in a criminal’s way of thinking.
If you don’t do this, you are trying to understand and explain the crime based on your logic.
Similarly, in order to “attract” bugs you have to wanna see problems; you have to identify problems that might bug several kinds of stakeholders; you have to put all your knowledge about the project in to consideration; and you have to be able to see the problems that matter. You are not trying to understand and explain the system by using your own logic, instead you are using several input sources to do this: logic, subjective thoughts, people’s skill levels, complexity of system, technology, etc.

So let me present my take on a definition of “Bug Magnet”:

Being a Bug Magnet = Able to foresee possible problems (or able to spot opportunities for things to go wrong), in context.

The most important thing here is the last two words “in context”. That is, even if you have all the bug taxonomies and oracles in the world to support you, you have to be able to understand what matters in this project.
Knowledge about “all common problems in .NET applications” can help you sometimes. Knowledge about “all common problems in .NET applications that developer X often produce”, is however much more useful.
Understanding what bugs our users is more helpful than knowing about what bugs users in general.
Knowing about problems with focus in Windows applications is one thing, finding these problems during testing is to be able to spot opportunities when they are presented to you in your context (see http://thetesteye.com/blog/2010/11/windows-focus/ ).

Now, back to the questions in the beginning of the post.

• Is it luck?
  — No, even if luck sometimes help. For others it is more of welcoming serendipity.
• Is it faith?
  — I don’t believe in faith.
• Is it an ability that some are born with and some aren’t?
  — Maybe. Some people might have a more developed talent, but I think that most people have this talent. Some are born with variations of narcissistic personality disorders and might have difficulties with this.
• Can you learn this ability?
  — I believe so.
• Can you improve it?
  — Yes. However, you might need to consider one or several dimensions to improve: Empathy, reasoning, attention for detail and seeing the whole, recognizing patterns of your own and other peoples mistakes, subjectivity,  general systems theory, context-driven testing, and more.
  Notice that these dimensions are not technical but rather comes from social sciences.