thoughts from the test eye » Ideas

The Scripted and Exploratory Testing Continuum

Henrik Emilsson — Thu, 05 Jan 2012 09:22:26 +0000

I have been using the Scripted – Exploratory Testing Continuum (For one source of this, see page 56 in http://www.ryber.se/wp-content/EssentialTestDesign.pdf ) in classes to explain how scripted testing and exploratory testing intertwines; and to explain that most of our testing is somewhere in the middle of the continuum.

But I have also had some issues with this description.

If this continuum exist, when can you say that you are doing more exploratory than scripted (and vice versa)?
How do you recognize when your testing is more exploratory than scripted (and vice versa)?

I do not believe that these questions can be answered because I think it has to do with the mindset of the person who sets the approach; and how far this particular mindset can be stretched.

The continuum is a simplified (yet useful) description of how your testing approach is a mix of the two approaches. And the simplification lies in that it can be seen as if our testing approach can be found in some point on the continuum. But I think that someone who has an exploratory testing approach is doing more or less exploratory testing; and someone who has a scripted testing approach is doing more or less scripted testing. Maybe this is how many of you already think of this?
However, I would like to visualize the differences in a slightly different way in order for me to understand this better.

Testing can be viewed from different point of views; and if I as a tester look at my testing approach I can have good use of the continuum. But the decision on which approach we should focus on is often driven by the management of the project or test team. So, from a managerial point of view, the testing approach is driven from a core dimension (interest, factor) which is more or less strict. I.e., I think that each approach can be seen as a set of continuums where each continuum consist of a certain dimension; and you will only modify these individual dimensions, not the entire approach.

Scripted Testing Continuums (some examples)

Exploratory Testing Continuums (some examples)

Let’s say that my main interest is Control, and I believe that a scripted testing approach is the best option in order to fulfill my interest. From this, if I now gradually become less stricter when it comes to my control demand, I won’t transit into an exploratory testing approach. My approach would instead be gradually less scripted.

In the same fashion, if my main interest is trust. My approach would gradually become less exploratory, but never transit into a scripted approach.

I think that this has to do with that the approach is a mindset; or rather an ideology. And it is hard for many people to change their mindset/ideology because your approach is derived from your school of thought.
If you have a scripted approach to testing, it is because of something you strongly believe in. If you have an exploratory approach to testing, it is because of something else you strongly believe in.

Pair testing with a flare of checking

Martin Jansson — Wed, 04 Jan 2012 12:23:08 +0000

When I explain pair testing I usually say that you pair up two persons. One drives the testing while the other documents, comes with suggestions and asks for clarifications.

In this context, if we consider the use of testing and checking. What if the person driving focus on testing (that is usually the case) and the other focus on his other tasks but also that of managing checks? Try to avoid disrupting the creative, exploratory mind of the tester driving the testing. Still, at some stages asking if it is possible to check certain things. I am sure a lot of persons work this way and it is nothing new, still perhaps they have not expressed it this way.

If you set it up this way you might avoid the various biases and blindness that the driver might experience. Instead you let the passive player focus on those things. By having a dialog you will naturally affect the other, but perhaps not as much as when you focus on it yourself.

What if you in a pair testing session, switch place every now and then. Each time you switch place you also switch from Test Driven to Check Driven.

What if you did pair testing but both of you tested in the same area at the same time with collaborative note taking. One had focus on Test Driven Checking while the other had Check Driven Testing. Would you get any interesting result based on that?

Some initial thoughts on checks

Martin Jansson — Sun, 01 Jan 2012 20:31:58 +0000

Introduction

In 2009 Michael Bolton had a talk at Agile 2009 from which he later on wrote about a distinction between testing and checking [1]. He has since then elaborated more in the area and digged deeper into it [2], and continue to do so [3]. By clarifying what we are doing at a certain time it makes it easier to determine what tools and heuristics to use among other things. By making this distinction I think it helps us as testers to do a better job and to better be able to explain to other non-testers why we need to do both testing and checking, but also what the difference is.

The idea that you can (and should) automate all testing is not interesting. It is like discussing automation of all decision making. In a discussion like this we can instead offer that checks are usually good to automate, but testing is not as easy since there is no true pass/fail result when it comes to open-ended information, that might have different value to different persons. James Bach makes an example how this is handled in BDD [4] or rather making it obvious that some checks might be easy to confirm while others are close to impossible.

When I talk about testing vs. checking and touch the subject about sapience [5], there are many who are known to promote the checking part as their main activity as a tester, who then feel uncomfortable. What is really important is to point out that they do exploratory testing as well [6], but perhaps not framing [7] or structuring it well. I’ve heard arguments that exploratory testing is less structured and formal, well perhaps for you it is. Why not change that?

Simon Morley makes some interesting observations in his post on Sapient checking [8] and I agree with him that it might confuse more if we talk about testing and checking when we are delivering information to inform decisions. Should the focus in our information be on if it was checking or testing being made? For most cases, I believe we should not.

Test Driven Checking

If we assume that testing and checking is intertwined, you might still consider that in some cases one thing drives the other or that at least one activity is in focus while the other is in the background.

Depending on how well I have organised my charters and missions for testing during a project I test differently. Sometimes when I run a test session (following principles from SBTM [9]) I do note taking on what tests and checks I have on the system, mixing the two activities. Afterwards I might go through a checklist and mark items off as done. This might be coverage related, requirements related or whatever was suitable to have in a checklist. I might have stored most of the information in my session notes, but I might still go into other areas to mark it there as well, if that was easily done. A college did exploratory testing, without note taking, and then went into an excel sheet to check off coverage areas. He then reported progress and coverage based on that.

To avoid missing some checks you might consider that after you have tested in an area and you are ready to move on, you take a peek at things to check in that area. By doing so you might avoid the confirmation bias [10] and unintentional blindness. You might also get new ideas for testing when seeing things that you should check.

When you do regression testing you might consider to first test around the area of the bug, then as a final step check if the reported bug is in fact fixed.

Check Driven Testing

This way of testing is probably the most common one by those who are unfamiliar with concepts of exploratory testing. As I see it, you might have a list of checks (or for some scripted tests) that you check, but you might test around them as well. In this case the checks are driving the testing. When you have one-liner test ideas as your main focus, I think it is most common that you let them drive you. So depending on how you define them you will have different focus. Simplifying it, are they “What if”-focused or “Verify that”-focused?

When you retest a fixed bug you might select the methd to check that the bug as it was reported is actually fixed, then you continue with what valuable testing around to see that nothing else has broken in the vicinity, as far as you know.

If you do a testing tour you might set out some designated targets (check points) and along the way do your testing reporting what you actually find out.

If you consider how kids might act when they walk, on their own, on one side of the street in one direction and all of a sudden they see their parents on the other side. The kid forgets everything they had learnt about traffic rules and risks, then just runs to the parents by crossing the street. I believe we can see this with testing and checks as well. When you get to the point where you can do the check, you forget all other things that you should really be testing, thus a case of unintentional blindness.

Conclusion

This is just one way of looking at it, this is no real conclusion. For me it helps to make a distinction between doing Test Driven Checking and Check Driven Testing. By being aware of the distinction you might plan your testing differently, you might consider a bit more what where you put your checks and where you do your testing. They will affect each other and they will be drawn to each other.

References

[1] http://www.developsense.com/blog/2009/08/testing-vs-checking/

[2] http://www.developsense.com/blog/2009/09/transpection-and-three-elements-of/

[3] http://www.developsense.com/blog/category/testing-vs-checking/

[4] http://www.satisfice.com/blog/archives/638

[5] http://www.satisfice.com/blog/archives/99

[6] http://www.developsense.com/blog/2011/05/exploratory-testing-is-all-around-you/

[7] http://www.developsense.com/blog/category/test-framing/

[8] http://testers-headache.blogspot.com/2009/09/sapient-checking.html

[9] http://www.satisfice.com/sbtm/index.shtml

[10] http://www.workroom-productions.com/papers/The%20Irrational%20Tester%20v1-06.pdf

Funcacles or unactivated oracles?

Henrik Emilsson — Tue, 15 Nov 2011 14:00:44 +0000

Last week I took the Rapid Test Management (RTM) class in Gothenburg with James Bach.
As always, he is such an inspiration and really makes you engage and think. Thanks for that James!

In one part of the class we were factoring an application:

Factoring: Dimensions of Interest
Factoring is the process of analyzing an object, event, or model to determine the elements that comprise it.
When we factor for the purposes of test design, we identify elements that we may need to observe or control during a test. We call those factors.
Factors may be
– intrinsic or relational
– variable or static
(quote from Bolton’s http://www.developsense.com/presentations/2010-04-ConfirmationBias.pdf)

At CAST 2011 I was taking Michael Bolton’s tutorial on Test Framing, where we discussed a lot about oracles. Michael is, like James, such an inspiration!
(The main thing for me with taking their classes is to get involved in their brains and then I know something will trigger in mine…)

But it was on the RTM class where I got to think of something that is missing here!

When we determine the elements that comprise an application we don’t really use oracles, since an oracle is a heuristic principle or mechanism by which someone recognizes a problem.
But we do use something similar to oracles when we determine the factors. It is like if you are equipped with a lot of oracles and they don’t get triggered – and this tells you that you that you have identified a function (that works).

Also, when factoring and modeling an application, I sometimes make qualified guesses on functionality that could be, or ought to be, present (indicated with a question mark in my model) but that I’m unable to identify at the moment; or it may also be implicit or unintentional functionality that I need to investigate further.

What heuristic principle or mechanism am I using to recognize a function?
What heuristic principle or mechanism am I using to recognize an ought-to-be function?
Since it cannot be an oracle (by definition), it must be something else.

My two suggestions are:

Funcacles (merged word that stands for Functionality Oracles)
Unactivated Oracles

Let me elaborate.
If you apply HICCUPS(F) on an application and use the consistency oracles – but they aren’t triggered, you have that heuristic principle that tells you if something works (or that might work; or that might be applicable).
Let’s have a look at a shortened description of HICCUPS(F):

Consistency is an important theme in oracles. Unless there is a compelling reason to desire otherwise, we generally tend to want a product to be consistent with:
History: That is, we expect the present version of the system to be consistent with past versions of it.
Image: We expect the system to be consistent with an image that the organization wants to project, with its brand, or with its reputation.
Comparable Products: We expect the system to be consistent with systems that are in some way comparable.
Claims: We expect the system to be consistent with what important people say about it.
Users’ Expectations: We expect the system to be consistent with some idea about what its users might want.
Product: We expect each element of the system to be consistent with comparable elements in the same system.
Purpose: We expect the system to be consistent with the explicit and implicit ways in which people might use it.
Standards and Statutes: We expect a system to be consistent with relevant standards or applicable laws.

There is one more heuristic that testers commonly apply. Unlike the preceding ones, this one is an inconsistency heuristic:
Familiarity: We expect the system to be inconsistent with any patterns of familiar problems.

(Shortened quotes from http://www.developsense.com/resources/Oracles.pdf)

I believe that when we are factoring and modeling we use all the consistency oracles above, but in an unactivated fashion. However, if we want to avoid using the word Oracle for this, then I suggest Funcacle in order not to confuse (sic!).

This is an important part of test analysis that I think should be elaborated further.
I’ll get back on this as soon as I have thought about it some more.

Do you have any suggestions?

Seventeen Test Ideas

Rikard Edgren — Mon, 17 Oct 2011 19:00:50 +0000

As an exercise, try these generic test ideas on any product, for instance a recent upload to sourceforge.net. Then come up with a better test idea, and write as a comment. Also suggest one to remove; I have decided they must be seventeen (a Tranströmer homage)

* Try to do what it is supposed to do
* Verify that Help matches functionality
* Let it run for quite some time, and look at resource utilization
* Play around with core functionality for five minutes
* Ask someone else for their first impression
* Perform any simple scenario as fast as possible
* Use keyboard-only for a while
* Have a look at all installed files
* Get rid of all traces of software (Uninstall)
* Run on a radically different platform, preferably an Error-Prone Machine
* Use credibly dirty data
* Provoke a bunch of failures
* Be a user that want to destroy for others
* Compare with common behavior for the environment
* Review About dialog information
* Is it at least as good as a comparable product?
* Do something valuable for you

Investigate your feelings during these tests; keep usability, charisma, security, performance and testability in the back of your head.
End by describing the state of the product in less than a minute.

These are test ideas you can try on any product, and they will give you interesting information.
However, spending the same amount of time on other tests, might be even better…
Really good test ideas are specific for a product, capturing their essence; challenging a main purpose.

It’s the little things

Martin Jansson — Sun, 25 Sep 2011 18:18:51 +0000

As a tester you find lots of things that bugs you when exploring a system. In some cases these issues only nudge you slightly at first, but after passing over the same issue many times it really starts to drive you crazy. This, at first small issue, has now become something that affect you more than you initially thought.

An example:
A feature in my Samsung mobile is annoying. I have a bluetooth headset. When I switch this on and enter the bluetooth settings on the mobile I cannot find the device the first time I enter or scan for it. I must go back then re-enter to find it. I do this every time I connect the bluetooth, which is a few times every day. After having the phone for a month it starts to irritate you. It might be my headset that is buggy or might be the way bluetooth is done on the Samsung. Either way, my experience of my Samsung is damaged and especially this issue is something that nudges me continuously.

When we are testing and report an issue like this, it is usually set to minor severity and as times goes by we want it to be of higher severity. As I see it, it is your obligation to update the bug with your new feelings and reactions. Because our first reaction to something might not be what we feel about the same subject further on. Our initial estimation on severity and priority of bugs should therefore be continuously revised and rethought, if time permits that is.

For many testers you will start to ignore the issue and not see it anymore, thus becoming biased [1]. There is also a big chance that you will suffer from the effect of the Broken Window Theory [2] by accepting similar issues to pass by. It is my belief that we very often give a faulty set of priority and severity when we do not know the system well, but the more we learn and understand the better chance we have to know how important and serious the bugs are.

References

[1] http://www.workroom-productions.com/papers/The%20Irrational%20Tester%20v1-06.pdf

[2] http://www.theatlantic.com/magazine/archive/1982/03/broken-windows/4465/

Testing Speeds Development

Rikard Edgren — Sun, 18 Sep 2011 10:57:27 +0000

This Wednesday I held a presentation at DevCon 2011 entitled Exploratory Test Design (slides)
I like this terminology a lot, because it encompasses the two things I want to see in software testing
* looking at a lot more information sources than requirements
* vary execution and look for many things

It was also a good presentation experience, because I could cherry pick my favorite heuristics, and talk freely about the ones I care most about, right now.
The audience was mainly programmers, and they are equally interested in exploratory testing and software potatoes!

But the reason I write this post is something Joakim Holm said at his talk, that should be told more often:
if you do a lot of testing, development goes faster

When testing provides continuous feedback, developers understand what is good, what is unreliable, and also what is important.
Continuous corrections towards the goal speed up the time to completion.
Testing isn’t a role with its own goal, we are there to help; and I think the most important help is speed, and the second one is quality (acceptable quality can be met in many ways, with testing it is faster.)

Someone might object and say “on the contrary, with all their phases and complaints, testing makes things much slower!” so I will rephrase:
good testing makes software projects go faster

Bug Magnets are thinking as criminals

Henrik Emilsson — Tue, 16 Aug 2011 04:17:15 +0000

I know of some testers who are pointed out by others to be Bug Magnets; people recognized for their ability to somehow draw bugs to them. Bug Magnets can be found in many workplaces and I bet that you know of someone that falls under this description. I have been appointed a Bug Magnet by some and it have made me thinking on what this phenomena boils down to.
Is it luck? Is it faith? Is it an ability that some are born with and some aren’t? Can you learn this ability? Can you improve it?
I have wondered about this for some time.

However, this summer I had a revelation when I watched an episode of the marvelous TV series “Homicide: Life on the Streets”:

Homicide: Life on the streets
Season 1 Episode 9 “Night of the dead living”
Det. Frank Pembleton and Det. Tim Bayliss

Bayliss [sits pensively. To Frank]: What are you looking at?

Pembleton is playing cat’s cradle.

Bayliss: You have something that you wanna say to me?

Pembleton: Adena Watson. So many unanswered questions.

Bayliss: And you’re saying that I’m not asking them.

Pembleton: I’m saying that you’re not answering them. [He peers at Tim through the cat's cradle.]

Bayliss [sighs]: What questions aren’t I answering, huh?

Pembleton [gets up and flips through a notebook, draws a picture]: Okay, these sixteen row houses on the north side of Kirk Avenue. Adena’s body was found outside the kitchen door in the red yard at 718 Kirk. Now the killer could have dropped her anywhere. Why not the common alley? Why not the yards at either end of the block? These three row houses are empty. One, two, three. The killer would have stood much less of a chance of being seen if he’d dumped her body in any one of these yards. Why would he risk bringing a little girl’s body inside a closed fence of an occupied house?
Maybe he wanted her body to be found immediately. Maybe he wanted to cast suspicion on the people in 718. Maybe he had some … perverse sense of remorse, some impulse to leave her body inside an enclosed yard to protect her from stray dogs.

Bayliss: These are *exactly* [taps the notebook] the questions that I have been trying to answer.

Pembleton: Well, you can try, but you never will.

Bayliss: Why?

Pembleton: You don’t think like a criminal. You don’t have a criminal’s mind.

Frank walks away. Tim grimaces in disbelief.

Bang! Suddenly it struck me. “You don’t think like a criminal. You don’t have a criminal’s mind”.
In order for a police to think of possible outcomes of a crime they have to be able to think as criminals, and put themselves in a criminal’s way of thinking.
If you don’t do this, you are trying to understand and explain the crime based on your logic.
Similarly, in order to “attract” bugs you have to wanna see problems; you have to identify problems that might bug several kinds of stakeholders; you have to put all your knowledge about the project in to consideration; and you have to be able to see the problems that matter. You are not trying to understand and explain the system by using your own logic, instead you are using several input sources to do this: logic, subjective thoughts, people’s skill levels, complexity of system, technology, etc.

So let me present my take on a definition of “Bug Magnet”:

Being a Bug Magnet = Able to foresee possible problems (or able to spot opportunities for things to go wrong), in context.

The most important thing here is the last two words “in context”. That is, even if you have all the bug taxonomies and oracles in the world to support you, you have to be able to understand what matters in this project.
Knowledge about “all common problems in .NET applications” can help you sometimes. Knowledge about “all common problems in .NET applications that developer X often produce”, is however much more useful.
Understanding what bugs our users is more helpful than knowing about what bugs users in general.
Knowing about problems with focus in Windows applications is one thing, finding these problems during testing is to be able to spot opportunities when they are presented to you in your context (see http://thetesteye.com/blog/2010/11/windows-focus/ ).

Now, back to the questions in the beginning of the post.

Is it luck?
– No, even if luck sometimes help. For others it is more of welcoming serendipity.
Is it faith?
– I don’t believe in faith.
Is it an ability that some are born with and some aren’t?
– Maybe. Some people might have a more developed talent, but I think that most people have this talent. Some are born with variations of narcissistic personality disorders and might have difficulties with this.
Can you learn this ability?
– I believe so.
Can you improve it?
– Yes. However, you might need to consider one or several dimensions to improve: Empathy, reasoning, attention for detail and seeing the whole, recognizing patterns of your own and other peoples mistakes, subjectivity, general systems theory, context-driven testing, and more.
Notice that these dimensions are not technical but rather comes from social sciences.

Closing note:
This is my response to one part of what I and Rikard have discussed during the last year: What constitutes skilled software testing?
More to come!

The Metrics Tumour

Rikard Edgren — Wed, 03 Aug 2011 06:28:50 +0000

quantitative numbers in a world of qualitative feelings

I am not against measurements in general, they can surely be useful. I use length when building things, weight for baking, time for appointments etc.
I often use numbers for various things in my bug reports.

But metrics are something different; metrics are measurement plus value.
“Should have at least 80% code coverage on unit tests” is a typical example, where “peer reviewed and accepted” would give better results.
“2% better defect detection percentage since last release!” says less than conversations with support department.
“95% Pass on test cases” means nothing at all.

The measurements with value cannot judge what is important;
reality is impossible to aggregate;
metrics are dangerous.

There are many good software testers that advocate metrics. At a couple of occassions I have had the chance to have “the talk” with some I respect.
It boils down to the same thinking: metrics must have a lot of context in order to be useful, so much details that I believe you could throw away the numbers, or only have them as a footnote.
This is not done, because management demands numbers, that’s what they are used to base decisions on.
But for software and testing that might not be well applicable.
If kids want candy, it doesn’t mean you have to give it to them.
You give them better things, out of respect and care.

Testing provides information, so you should find out what is important to different people, and give them what they need, not necessarily what they want.
Metrics will hide importance, and also skew the development efforts.
Try to take them out, with surgical precision.

Then you are left with one final catch:
Qualitative information is hard to aggregate, trust is needed.

Story telling and week[end|night] testing

Martin Jansson — Thu, 21 Jul 2011 06:33:21 +0000

Story telling is an important part of testing. It is a part where you communicate and tell a compelling story what information you found at. Each story needs a scene in which it plays.

Some of you might have attended the weekend or weeknight test sessions, some might have attended classes in testing where they teach by painting you a scene in which you act. For those of you who are familiar with roleplaying games or story telling games, this is just like the story teller describing the scene and where you as a character act in it as well as interact with other characters. In order to learn the most and to be able to act the best of your abilities, the story teller need to paint the scene so vivid that you can imagine it yourself. The characters you interact with also need to be full of details, have an history and have agendas of their own.

When I took the class in Rapid Software Testing by James Bach, he had several exercises where he painted scenes that we interacted in. Some in the class were very unused to this, but James played the story well and coached them into acting. The best exercises were those that he had been practicing a long time that had lots of details and where you interacted with many different characters. Each character had a history of their own and he could tie them to individuals that he himself had interacted with in past projects. James laid out lots of traps that you were supposed to avoid or escape. He sometimes named what you did so that you can use it as an ability or heuristic in the future and more easily talk about what it was.

Some of you might be experienced with MUD:s, MUSH or similar. These are text based games that sometimes have a flare of role playing in them. As a Wizard (a programmer or game master) you sometimes had tools to you could utilize to make the scenes more vivid. You added *feelings* and smileys to your regular text messages to make a distinction between your moods. This has become a natural part of our language when chatting. Still, you can use it even more vividly when story telling in a chat.

How can you then apply your story telling skill to the test session? Here are a few ideas:

Paint the scene well before the session, with all characters, the mission etc
As a group of participants, what is your role? Are you a team? Do you know each other from before? How were you gathered? If there is a mix of skill and experience, how do you explain that? How do you intend to act it out? If you are experienced will you act in a different way so that the traps laid out will give a better learning experience for the in-experienced ones? It is good if this is defined before the session start.
Put more details in the characters. Let them have names, titles and a brief history. It will make the decision making, interaction and role playing in the scene alot better.
As a participant, when asking questions be clear to whom of the characters you talk to.
As a session leader, when asked questions be clear on which character that answer.
As a session leader, be clear when you want to give out general information about the scene, information about the characters in it or things that the characters notice as one “voice”.
As a session leader, use the power of story telling such as “Meanwhile somewhere else … ” where you paint a scene to show things that happen outside the scope of the camera. This is a good trick to bring even more detail and power to the story being played out. Visualizing what you do with the camera such as zooming in or out can also paint a more interesting scene.
As a session leader, do you want someone to act test lead so that they can practise leadership and organisational skills? If so, perhaps coach them into the story and scene a bit before hand.
When chatting add feelings and moods to your regular chat message so that you can roleplay differences in what you say, as hidden messages between the lines.

I understand that there is a limited time issue both for preparation and when actually doing the session, but doing some of these things might help to some extent. Who knows, you might learn more?