On 7-9 May the Let’s Test Conference will take place. During the day there will be lots of interesting tutorials, keynotes and sessions. During the evening the events will continue. One of these activities is the Testlab, that we call Let’s TestLab. Initially I misunderstood Henrik Emilsson when we started to organise the lab. I thought the evening event was the testlab. At the time I did not consider anything wrong with having 150 people or more in the testlab. As I saw the evening event program I considered how could I compete with such a fine setup of activities. Well, this is a conference with many context-driven people. We will have different interests, expectations and focus area. So instead of admiting defeat, I considered what elements that we could add to the testlab to bring a great crowd, but perhaps not all of them.

Here is the line up of events in the testlab:

• Collaborative test planning
• Group Test Experiment
• Test Competition
• Making a compelling status report

Collaborative test planning

We will create possible charters, missions for the coming testing in the lab so that those who wish can practise different testing techniques. Everyone is invited to share his or her idea on how to plan testing in through collaboration.

Part of the line up in this event is:

• Rikard Edgren
• Christin Wiedemann

Group Test Experiment

The context of the testlab will be single testers or groups of testers going in and out of the testlab in different time intervals. Each tester will be unique in the sense that they bring different level of experience, skills and approaches to testing. Based on this, we will start experimenting with group testing. I think we have limited ourselves too long with the setup of pair testing. Going back to the early recommendations and experiences from Brian Marick, James Bach, Cem Kaner, Jonathan Kohl and many others, the setup is nearly the same. With new tools and techniques appearing over the years, some assumptions could be questioned.

Let us assume that there are different aspects and combinations of group testing that serves different purposes, we can say that there are different dimensions that could be explored.

Here are a few dimensions that we will experimented with:

• How many testers (2 or more)
• What role you play as a tester
• User types, User Scenarios or storytelling
• Mission of group test
• Note taking techniques
• Partner combination
• Lateral thinking aspects
• Personality types
• Debriefing techniques
• Accountability
• Focus areas or Characteristic focus
• Test Environment
• Basic Configuration Matrix
• … and new ones that we find along the way

We will explore and experiment with different tools that we use when group testing and share experience on what works in what context. We also experiment with a few pre-defined group test setups such as:

• Cinema testing
• 15-min test run
• Coaching a group of testers
• Wolf pack concept
• Testing Dojo

Part of the line up in the test lab is:

• Ann-Marie Charrett
• Markus Gärtner and Meke Mertsch
• Johan Åtting

Test Competition

Can you really compete in testing? Can you compete between two or more teams?  Can you really estimate the value of one piece of information against another? Well, it depends.

I have been an Ultimate Frisbee player for 34 years. I’ve not played for some time, but once a frisbee fan, always a fan. I think the same goes for testing. There are many things that I feel is similar. Craftmanship/sportmanship and passion is major part. Here is one description I like:

Ultimate has traditionally relied upon a spirit of sportsmanship which places the responsibility for fair play on the player. Highly competitive play is encouraged, but never at the expense of the bond of mutual respect between players, adherence to the agreed upon rules of the game, or the basic joy of play. Protection of these vital elements serves to eliminate adverse conduct from the Ultimate field. Such actions as taunting of opposing players, dangerous aggression, intentional fouling, or other “win-at-all-costs” behavior are contrary to the spirit of the game and must be avoided by all players.

One important element in Ultimate Frisbee is the spirit of the game, which you can see more in detail here [1]. Passion and humility as a tester are important traits, the spirit of the game concept might help us here.

So, my idea for a tester competition will be based on some of these ideas. Two teams compete against each other in form of best bugs and session notes. The two teams go through the opposite teams material and conclude who they think should be the winner with a good reason why.

Make a compelling status report

As the last event in the testlab we want to investigate how we can make a compelling status report for our stakeholders. Having many different testers, session notes spread all over, half-finished bug reports and test ideas half-finished… can we create something that is still valuable to someone? I guess this is a common situation at any lab at any company, still we will dig deep into how to go about this.

Part of the line up in the test lab is:

•  Ben Kelly

References

[1] Ultimate Frisbee Rules –  http://www.cs.rochester.edu/u/ferguson/ultimate/ultimate-simple.html

Scripted testing vs exploratory testing approach

I agree with the idea of a polarization between the scripted test approach and exploratory test approach. These approaches include how you perceive testing and a tester. Almost in the same sentence, some say that you do a bit of both scripted and exploratory testing. The perception on testing and how you conduct testing are two different things. I believe that by discussing them together and especially with the polarization example, it more than often confuse the listener.

Instead of saying that you do both scripted and exploratory testing, I think it is more fruitful to talk about testing and checking. M. Bolton and James Bach states that “A check is a component of a confirmatory approach to testing.” in the blog article Elements of Testing and Checking [1]. We might even be so bold to say that it is the main component, but then we might say that another part is smaller and that might not really be true.

The Checklist

One aspect of checks could be something similar to what the co-pilot does when assisting the pilot. If we see the pilot as the explorer, reacting to input and performing based on the current context. In certain situations the co-pilot brings up checklists to help go through a situation that is too complex to remember all the details about. It is up to the pilot and co-pilot to know when the checks are relevant or not. The same situation can be found at the operating table where you have several types assisting personnel that help perform a complex surgery. The use of a checklist in those cases would help avoid the easiest mistakes or misunderstandings. See The Checklist Manifesto [2] for more ideas for testing.

How does this apply to testing then? When we do pair testing there is usually one driver while the other is a bit more passive and documents. What if the driver is the pilot/explorer and the other is the co-tester handling documentation, checklists and checks? When doing collaborative test planning, group exploratory testing or other collaborative test activities, do we then split up in roles such as tester/explorer, co-tester, checker etc to help us define what we do and focus on? Does it provide value to do so?

Types of Checks

Here is a definition of a check [1]:

a check itself has three elements:
1) It involves an observation.
2) The observation is linked to a decision rule.
3) Both the observation and the decision rule can be performed without sapience (that is, without a human brain).

But could we split checks in different parts? What types of checks are there? M. Bolton describes questions that has to do with confirmation, verification, etc. Could we also break down checks into other categories such as checklists, test/quality patterns, guided checks and one-liner checks.

A checklist is a list of items that help you through a complex situation. The items in the checklist could be checks that need to be confirmed or verified to a certain extent.
A test/quality pattern is something that repeatedly gives suspicion that it might be broken so that you need to check it.
A guided check is what we previously called a scripted test. A set of steps that end with something that you wish to verify or confirm, something with a binary answer.
A one-liner check or a check idea is a brief statement of something that should be checked in a certain context that would give a binary answer, such as true/false, ok/nok or yes/no.

Checking clarifies an aspect of what testing is or is not, do these sub-categories help us clarify what we do with the checks? Could we find more types of checks?

Test Management Tools

Based on the assumption that there are at least two types of questions and at least two different types of answers, how do we structure and manage these today? The traditional management tools for testing can often handle all the above categories for checks, but they are unable to handle the answers from testing.

But what do we mean when we say handle in this case? Well, you can structure the checks, plan when you do them and report the result of them. You can then make reports on the overall progress and the overall quality of the system. You can also state what build, system or setup you used in the test. But since the tools cannot really handle the testing part, the progress and the picture on quality really becomes obsolete.

If we look at tools and techniques that are more aligned with testing such as SBTM, they are good at handling the testing story. They are not as good at handling checks, or rather the tools I have seen so far.

Can we and do we want to handle the result from our testing and checking in the same system? When working in small teams where there are fewer stakeholders to work for, the need for information sharing in a big system can be less important. But if you are working in an organization where you have teams in several countries and where the overall development organization can be thousands of people, it can be more important to share information in a bigger system. Still, are you actually able to share information to that many people in an efficient way just because you use an advanced management system? When a lot of people are involved in creating and sharing information there is also a bigger chance that the actual meaning is lost. You then oversimplify the situation that information sharing is easy. If the organisation is big, there is a big chance that the context changes over the organisation and that the information changes meaning over the organisation.

I’ve seen so many test management tools that try to solve the whole test process. By focusing on all aspects they also become quite crappy at all aspects. Picking the tools that are excellent at solving one problem can then be a lot more efficient even if you get to work with several tools. When working with test coverage, communicating your test ideas, reporting status, reporting progress and showing details on what you tested there is a big chance that one tool really cannot solve this for you. We know that we find new techniques from other disciplines that can solve a problem for us when testing. So why do many limit themselves to test case management systems?

Summary

Michael Bolton and many other great minds have explored and delved into this concept, I think we can find more pieces that we can shed some light on. I’ve identified a few areas and there are more to come.

References

[1] Elements of Testing and Checking – http://www.developsense.com/blog/2009/09/elements-of-testing-and-checking/
[2] The Checklist Manifesto – How to Get Things Right - http://www.amazon.com/The-Checklist-Manifesto-Things-Right/dp/0805091742

I´m sure you have heard it before. Everyone can test or Everyone does testing. Is that so? Is that really the case? Do you test just because you use a product? Do you test just because you stumble upon a bug? Do you test just because you can write some detailed step into a test management tool?

A while ago I was going to a customer meeting to hold a workshop in SBTM, showing that testing could be managed in a different way. I feel fairly experienced in the way I work as a tester, but at a parking lot my humility and confidence turned over.

I was standing next in line to pay for the parking ticket. The middle-aged man in front of me was having trouble to get a ticket. He thought it might be something wrong with his credit card, so I offered to try out my card to test if it was the card or if the machine was in fact broken. There were several people behind me. After noticing that my card didn’t work either, which I knew had worked earlier the same morning, we concluded as a group that the parking machine was broken. There was another machine a bit further away that seemed to work, based on that the people using it were getting parking tickets and using their cards, so all of us went there.

On the way back to my car I walked past, what we called, the broken parking machine and two very old ladies were trying use it. I told the ladies that the machine was broken and they responded, “Yes, but how broken is it?”. I realized my blindness and my narrow definition of broken. Then the ladies said, “We are going to test to see if coins work.”. I rarely have any coins or paper money for that matter, so my narrowness on the test scope based on my own context of use, what I could do at that time.

I thought I had found a dead horse, but it was infact an imaginary one.

I have been using the Scripted – Exploratory Testing Continuum (For one source of this, see page 56 in http://www.ryber.se/wp-content/EssentialTestDesign.pdf ) in classes to explain how scripted testing and exploratory testing intertwines; and to explain that most of our testing is somewhere in the middle of the continuum.

But I have also had some issues with this description.

If this continuum exist, when can you say that you are doing more exploratory than scripted (and vice versa)?
How do you recognize when your testing is more exploratory than scripted (and vice versa)?

I do not believe that these questions can be answered because I think it has to do with the mindset of the person who sets the approach; and how far this particular mindset can be stretched.

The continuum is a simplified (yet useful) description of how your testing approach is a mix of the two approaches. And the simplification lies in that it can be seen as if our testing approach can be found in some point on the continuum. But I think that someone who has an exploratory testing approach is doing more or less exploratory testing; and someone who has a scripted testing approach is doing more or less scripted testing. Maybe this is how many of you already think of this?
However, I would like to visualize the differences in a slightly different way in order for me to understand this better.

Testing can be viewed from different point of views; and if I as a tester look at my testing approach I can have good use of the continuum. But the decision on which approach we should focus on is often driven by the management of the project or test team. So, from a managerial point of view, the testing approach is driven from a core dimension (interest, factor) which is more or less strict. I.e., I think that each approach can be seen as a set of continuums where each continuum consist of a certain dimension; and you will only modify these individual dimensions, not the entire approach.

Scripted Testing Continuums (some examples)

Exploratory Testing Continuums (some examples)

Let’s say that my main interest is Control, and I believe that a scripted testing approach is the best option in order to fulfill my interest. From this, if I now gradually become less stricter when it comes to my control demand, I won’t transit into an exploratory testing approach. My approach would instead be gradually less scripted.

In the same fashion, if my main interest is trust. My approach would gradually become less exploratory, but never transit into a scripted approach.

I think that this has to do with that the approach is a mindset; or rather an ideology. And it is hard for many people to change their mindset/ideology because your approach is derived from your school of thought.
If you have a scripted approach to testing, it is because of something you strongly believe in. If you have an exploratory approach to testing, it is because of something else you strongly believe in.

When I explain pair testing I usually say that you pair up two persons. One drives the testing while the other documents, comes with suggestions and asks for clarifications.

In this context, if we consider the use of testing and checking. What if the person driving focus on testing (that is usually the case) and the other focus on his other tasks but also that of managing checks? Try to avoid disrupting the creative, exploratory mind of the tester driving the testing. Still, at some stages asking if it is possible to check certain things. I am sure a lot of persons work this way and it is nothing new, still perhaps they have not expressed it this way.

If you set it up this way you might avoid the various biases and blindness that the driver might experience. Instead you let the passive player focus on those things. By having a dialog you will naturally affect the other, but perhaps not as much as when you focus on it yourself.

What if you in a pair testing session, switch place every now and then. Each time you switch place you also switch from Test Driven to Check Driven.

What if you did pair testing but both of you tested in the same area at the same time with collaborative note taking. One had focus on Test Driven Checking while the other had Check Driven Testing. Would you get any interesting result based on that?

Last week I took the Rapid Test Management (RTM) class in Gothenburg with James Bach.
As always, he is such an inspiration and really makes you engage and think. Thanks for that James!

In one part of the class we were factoring an application:

Factoring: Dimensions of Interest
Factoring is the process of analyzing an object, event, or model to determine the elements that comprise it.
When we factor for the purposes of test design, we identify elements that we may need to observe or control during a test. We call those factors.
Factors may be
– intrinsic or relational
– variable or static
(quote from Bolton’s http://www.developsense.com/presentations/2010-04-ConfirmationBias.pdf) 

At CAST 2011 I was taking Michael Bolton’s tutorial on Test Framing, where we discussed a lot about oracles. Michael is, like James, such an inspiration!
(The main thing for me with taking their classes is to get involved in their brains and then I know something will trigger in mine…)

But it was on the RTM class where I got to think of something that is missing here!

When we determine the elements that comprise an application we don’t really use oracles, since an oracle is a heuristic principle or mechanism by which someone recognizes a problem.
But we do use something similar to oracles when we determine the factors. It is like if you are equipped with a lot of oracles and they don’t get triggered – and this tells you that you that you have identified a function (that works).

Also, when factoring and modeling an application, I sometimes make qualified guesses on functionality that could be, or ought to be, present (indicated with a question mark in my model) but that I’m unable to identify at the moment; or it may also be implicit or unintentional functionality that I need to investigate further.

What heuristic principle or mechanism am I using to recognize a function?
What heuristic principle or mechanism am I using to recognize an ought-to-be function?
Since it cannot be an oracle (by definition), it must be something else.

My two suggestions are:

• Funcacles (merged word that stands for Functionality Oracles)
• Unactivated Oracles

Let me elaborate.
If you apply HICCUPS(F) on an application and use the consistency oracles – but they aren’t triggered, you have that heuristic principle that tells you if something works (or that might work; or that might be applicable).
Let’s have a look at a shortened description of HICCUPS(F):

As an exercise, try these generic test ideas on any product, for instance a recent upload to sourceforge.net. Then come up with a better test idea, and write as a comment. Also suggest one to remove; I have decided they must be seventeen (a Tranströmer homage)

* Try to do what it is supposed to do
* Verify that Help matches functionality
* Let it run for quite some time, and look at resource utilization
* Play around with core functionality for five minutes
* Ask someone else for their first impression
* Perform any simple scenario as fast as possible
* Use keyboard-only for a while
* Have a look at all installed files
* Get rid of all traces of software (Uninstall)
* Run on a radically different platform, preferably an Error-Prone Machine
* Use credibly dirty data
* Provoke a bunch of failures
* Be a user that want to destroy for others
* Compare with common behavior for the environment
* Review About dialog information
* Is it at least as good as a comparable product?
* Do something valuable for you

Investigate your feelings during these tests; keep usability, charisma, security, performance and testability in the back of your head.
End by describing the state of the product in less than a minute.

These are test ideas you can try on any product, and they will give you interesting information.
However, spending the same amount of time on other tests, might be even better…
Really good test ideas are specific for a product, capturing their essence; challenging a main purpose.

As a tester you find lots of things that bugs you when exploring a system. In some cases these issues only nudge you slightly at first, but after passing over the same issue many times it really starts to drive you crazy. This, at first small issue, has now become something that affect you more than you initially thought.

An example:
A feature in my Samsung mobile is annoying. I have a bluetooth headset. When I switch this on and enter the bluetooth settings on the mobile I cannot find the device the first time I enter or scan for it. I must go back then re-enter to find it. I do this every time I connect the bluetooth, which is a few times every day. After having the phone for a month it starts to irritate you. It might be my headset that is buggy or might be the way bluetooth is done on the Samsung. Either way, my experience of my Samsung is damaged and especially this issue is something that nudges me continuously.

When we are testing and report an issue like this, it is usually set to minor severity and as times goes by we want it to be of higher severity. As I see it, it is your obligation to update the bug with your new feelings and reactions. Because our first reaction to something might not be what we feel about the same subject further on. Our initial estimation on severity and priority of bugs should therefore be continuously revised and rethought, if time permits that is.

For many testers you will start to ignore the issue and not see it anymore, thus becoming biased [1]. There is also a big chance that you will suffer from the effect of the Broken Window Theory [2] by accepting similar issues to pass by. It is my belief that we very often give a faulty set of priority and severity when we do not know the system well, but the more we learn and understand the better chance we have to know how important and serious the bugs are.

References

[1] http://www.workroom-productions.com/papers/The%20Irrational%20Tester%20v1-06.pdf

[2] http://www.theatlantic.com/magazine/archive/1982/03/broken-windows/4465/