thoughts from the test eye

Last week I took the Rapid Test Management (RTM) class in Gothenburg with James Bach.
As always, he is such an inspiration and really makes you engage and think. Thanks for that James!

In one part of the class we were factoring an application:

Factoring: Dimensions of Interest
Factoring is the process of analyzing an object, event, or model to determine the elements that comprise it.
When we factor for the purposes of test design, we identify elements that we may need to observe or control during a test. We call those factors.
Factors may be
– intrinsic or relational
– variable or static
(quote from Bolton’s http://www.developsense.com/presentations/2010-04-ConfirmationBias.pdf) 

At CAST 2011 I was taking Michael Bolton’s tutorial on Test Framing, where we discussed a lot about oracles. Michael is, like James, such an inspiration!
(The main thing for me with taking their classes is to get involved in their brains and then I know something will trigger in mine…)

But it was on the RTM class where I got to think of something that is missing here!

When we determine the elements that comprise an application we don’t really use oracles, since an oracle is a heuristic principle or mechanism by which someone recognizes a problem.
But we do use something similar to oracles when we determine the factors. It is like if you are equipped with a lot of oracles and they don’t get triggered – and this tells you that you that you have identified a function (that works).

Also, when factoring and modeling an application, I sometimes make qualified guesses on functionality that could be, or ought to be, present (indicated with a question mark in my model) but that I’m unable to identify at the moment; or it may also be implicit or unintentional functionality that I need to investigate further.

What heuristic principle or mechanism am I using to recognize a function?
What heuristic principle or mechanism am I using to recognize an ought-to-be function?
Since it cannot be an oracle (by definition), it must be something else.

My two suggestions are:

• Funcacles (merged word that stands for Functionality Oracles)
• Unactivated Oracles

Let me elaborate.
If you apply HICCUPS(F) on an application and use the consistency oracles – but they aren’t triggered, you have that heuristic principle that tells you if something works (or that might work; or that might be applicable).
Let’s have a look at a shortened description of HICCUPS(F):

I spent two days in Malmö attending a developer conference with a fantastic test track (put together by Sigge Birgisson.)

I did a presentation on Curing Our Binary Disease (slides, abstract), which was much better received than I hoped for (I thought it was a binary love/hate talk)
Good questions and talk about being inside the potato, how do you know where you are?

Pradeep and Rikard in front of Øredev projector

Pradeep Soundararajan talked convincingly about caring for the users, e.g. by using Twitter as a source for test ideas.
He also said the context-driven community has moved from Pass/Fail to “Is there a problem here?” (still a binary question though…)

Shmuel Gershon shared an experience report of a 100% exploratory testing project, with qualitative reporting and everything (“it’s called quality report, not quantity report”.)

Gojko Adzic talked about testers needing to adapt to how development is done now, and put most of their time ensuring others write good automated tests.
“There’s so much mistrust in the processes.”

Henrik Andersson wants, with right, a diversified testing team, and therefore only hires context-driven testers
Just joking, he said people can read the same excellent books, as long as they think critically and learn other things.

Selena Delesie explained how to focus the testing effort on customer needs; “testers are information radiators”.

Zeger van Hese made the presentation that I enjoyed the most. A thoughtful walkthrough similarities between the fine arts and testing.
“The Hungry Eye – thoughtfully looking at software”

Janet Gregory highlighted five enlightened areas since Agile Testing was released: feature acceptance, collaborative automation, large organizations, distributed teams, continuous learning.

Outside the venue I had many interesting conversations, there were testing games, food and drinks, Black Viper and even imitations, and all in all a very nice experience.
Thumbs up for a testing session with Pradeep and Shmuel that showed (the need for) different note taking styles.

The presentations will be made available online at www.oredev.org/

A year has passed since we released version 1.0 of our quality model without metrics.
It is time for a new version, with additions and corrections we have learned over the year (and a Swedish translation!)

1.1 English
1.1 Swedish
1.0 English

Feedback is always welcome!

/Rikard, Henrik, Martin

As an exercise, try these generic test ideas on any product, for instance a recent upload to sourceforge.net. Then come up with a better test idea, and write as a comment. Also suggest one to remove; I have decided they must be seventeen (a Tranströmer homage)

* Try to do what it is supposed to do
* Verify that Help matches functionality
* Let it run for quite some time, and look at resource utilization
* Play around with core functionality for five minutes
* Ask someone else for their first impression
* Perform any simple scenario as fast as possible
* Use keyboard-only for a while
* Have a look at all installed files
* Get rid of all traces of software (Uninstall)
* Run on a radically different platform, preferably an Error-Prone Machine
* Use credibly dirty data
* Provoke a bunch of failures
* Be a user that want to destroy for others
* Compare with common behavior for the environment
* Review About dialog information
* Is it at least as good as a comparable product?
* Do something valuable for you

Investigate your feelings during these tests; keep usability, charisma, security, performance and testability in the back of your head.
End by describing the state of the product in less than a minute.

These are test ideas you can try on any product, and they will give you interesting information.
However, spending the same amount of time on other tests, might be even better…
Really good test ideas are specific for a product, capturing their essence; challenging a main purpose.

Storytelling has been rising for quite some years and it will soon boom for software testing.
The reason is simple: people like stories.
And if it is used as status reporting instead of lame numbers, it is a step in the right direction, to say the least.
But when testing this idea theoretically, I find some fears:
* they will take long time to tell
* they will be too entertaining, less informative

And when comparing with my experiences, I find success stories when remembering conversations where I fast communicated the essence.

We already have a core tester skill in writing effective bug titles.
We should learn how to do this for whole test efforts, we should always have an executive summary up our sleeve.
We shouldn’t tell stories, it should be more like a librarian’s summary.

To accomplish this we need a lot of training, but I doubt it will be enough.
We also need more words.
They don’t have to be unanimously defined, and they can be reconstructed at each company, as long as they are useful.
Do a daily exercise of explaining the state of the product in thirty seconds, and try out your own terminology.

We need more words to effectively communicate the essence of our findings.

As a tester you find lots of things that bugs you when exploring a system. In some cases these issues only nudge you slightly at first, but after passing over the same issue many times it really starts to drive you crazy. This, at first small issue, has now become something that affect you more than you initially thought.

An example:
A feature in my Samsung mobile is annoying. I have a bluetooth headset. When I switch this on and enter the bluetooth settings on the mobile I cannot find the device the first time I enter or scan for it. I must go back then re-enter to find it. I do this every time I connect the bluetooth, which is a few times every day. After having the phone for a month it starts to irritate you. It might be my headset that is buggy or might be the way bluetooth is done on the Samsung. Either way, my experience of my Samsung is damaged and especially this issue is something that nudges me continuously.

When we are testing and report an issue like this, it is usually set to minor severity and as times goes by we want it to be of higher severity. As I see it, it is your obligation to update the bug with your new feelings and reactions. Because our first reaction to something might not be what we feel about the same subject further on. Our initial estimation on severity and priority of bugs should therefore be continuously revised and rethought, if time permits that is.

For many testers you will start to ignore the issue and not see it anymore, thus becoming biased [1]. There is also a big chance that you will suffer from the effect of the Broken Window Theory [2] by accepting similar issues to pass by. It is my belief that we very often give a faulty set of priority and severity when we do not know the system well, but the more we learn and understand the better chance we have to know how important and serious the bugs are.

References

[1] http://www.workroom-productions.com/papers/The%20Irrational%20Tester%20v1-06.pdf

[2] http://www.theatlantic.com/magazine/archive/1982/03/broken-windows/4465/

James Bach and Michael Bolton has a classic collection of consistency oracles; HICCUPPS(F): History, Image, Comparable Products, Claims, User Expectations, Product, Purpose, Standards and Statutes, Familiarity

It is a very good collection; not only helping you find out if something is a problem or not, but also the other way round: serving as testing inspiration.
At EuroSTAR 2009 tutorial Exploratory Testing Masterclass, Michael said that anyone coming up with additions would become famous.
As a tester this was of course an interesting challenge!
So over time, I now and then have thought of additions, but they haven’t been solid.
For example; V as in Violation of Quality Characteristics, could be seen as an inverted Familiar Problems or User Expectations.
You, as in the tester’s subjectivity can also be part of F and U.

But finally I found the important oracle missing from their list.
C as in Conversations.
You talk to a developer or other stakeholder, and collaborate to reach a decision.
Yes, during the discussion you might use HICCUPPS(F) oracles, but not necessarily. And it might not be the whole story, and not your oracle.
It might not be a 100% consistency heuristic, but is important enough to not leave out.
C as in Consistent with Conclusions from Collaborative Communication.

Outside the list we have the important No Oracle, where judgment is suspended, and noteworthy information is communicated, without deciding if it’s a problem or not.
But that doesn’t make for a nice soccer-sounding mnemonic, and is better left as a side-note to HICCUPPS F.C. heuristic.

(When double-checking, I noticed that they recently have added another one in the RST slides: Explainability – The system is consistent with my ability to explain it.
For now, I’ll put that one as a sub-part of Conversations; in dialogue, we weren’t able to explain the behaviour.)

History - the product should be consistent with previous versions
Image - consistency with the looks and behavior expected of the product
Comparable Products - consistency with competitors, references or other solutions to similar problems
Claims - consistent with written or oral statements about the product
User Expectations - consistent with what a reasonable user would expect
Product - consistent within itself
Purpose - consistent with the reasons for making/using the product
Standards and Statutes – consistent with standards or other authorities
Familiarity - inconsistent with problems seen before
Conversations - consistent with conclusions from dialogue

This Wednesday I held a presentation at DevCon 2011 entitled Exploratory Test Design (slides)
I like this terminology a lot, because it encompasses the two things I want to see in software testing
* looking at a lot more information sources than requirements
* vary execution and look for many things

It was also a good presentation experience, because I could cherry pick my favorite heuristics, and talk freely about the ones I care most about, right now.
The audience was mainly programmers, and they are equally interested in exploratory testing and software potatoes!

But the reason I write this post is something Joakim Holm said at his talk, that should be told more often:
if you do a lot of testing, development goes faster

When testing provides continuous feedback, developers understand what is good, what is unreliable, and also what is important.
Continuous corrections towards the goal speed up the time to completion.
Testing isn’t a role with its own goal, we are there to help; and I think the most important help is speed, and the second one is quality (acceptable quality can be met in many ways, with testing it is faster.)

Someone might object and say “on the contrary, with all their phases and complaints, testing makes things much slower!” so I will rephrase:
good testing makes software projects go faster

During my first paternity leave I learned sourdough baking. During the second I couldn’t help writing an ambitious paper, or a small book, about people-oriented test design, about things beyond test design techniques, close to the exploratory testing tradition.

It can be downloaded here.

It contains collections of knowledge, and generalizations of my ten years of testing the same product suite. I think it can be useful for ambitious testers that want to find any problems that might be important.

It probably is too much, theoretical, irrelevant or condense for many of you, but if you want to give it a shot I recommend the following:

* Download The Little Black Book on Test Design
* Print as double-sided A5 Booklet
* Find a quiet, comfortable place
* Read and relate to your test reality

Comments are welcome, especially additions to the collection of one hundred and three test design heuristics.

How do you find out what is important, in your specific situation?

I think it is the essential problem for all activities with complexity.
I think it is impossible to do really good software testing without the ability to dismiss things as not important, and dig deeper for matters that are important (I believe it’s the same for cooking a meal, or raising a child.)
I don’t think you can sample appropriately unless you know a lot more than the requirements.
I think we need better understanding and models for this, if we are to persuade those that want to govern by number; they aren’t hitting the crucial things.

It doesn’t seem like pointing to intuition and subjectivity is enough, but I haven’t reached further than this:

Either you know what is important (when you see it), or you don’t.
So the more knowledge you have about things that matter, the better suited are you to find important quality-related information; some as planned, some by serendipity.
So the key is to learn a lot, from a variety of sources.

I obviously need help on this one…