thoughts from the test eye

Scripted testing vs exploratory testing approach

I agree with the idea of a polarization between the scripted test approach and exploratory test approach. These approaches include how you perceive testing and a tester. Almost in the same sentence, some say that you do a bit of both scripted and exploratory testing. The perception on testing and how you conduct testing are two different things. I believe that by discussing them together and especially with the polarization example, it more than often confuse the listener.

Instead of saying that you do both scripted and exploratory testing, I think it is more fruitful to talk about testing and checking. M. Bolton and James Bach states that “A check is a component of a confirmatory approach to testing.” in the blog article Elements of Testing and Checking [1]. We might even be so bold to say that it is the main component, but then we might say that another part is smaller and that might not really be true.

The Checklist

One aspect of checks could be something similar to what the co-pilot does when assisting the pilot. If we see the pilot as the explorer, reacting to input and performing based on the current context. In certain situations the co-pilot brings up checklists to help go through a situation that is too complex to remember all the details about. It is up to the pilot and co-pilot to know when the checks are relevant or not. The same situation can be found at the operating table where you have several types assisting personnel that help perform a complex surgery. The use of a checklist in those cases would help avoid the easiest mistakes or misunderstandings. See The Checklist Manifesto [2] for more ideas for testing.

How does this apply to testing then? When we do pair testing there is usually one driver while the other is a bit more passive and documents. What if the driver is the pilot/explorer and the other is the co-tester handling documentation, checklists and checks? When doing collaborative test planning, group exploratory testing or other collaborative test activities, do we then split up in roles such as tester/explorer, co-tester, checker etc to help us define what we do and focus on? Does it provide value to do so?

Types of Checks

Here is a definition of a check [1]:

a check itself has three elements:
1) It involves an observation.
2) The observation is linked to a decision rule.
3) Both the observation and the decision rule can be performed without sapience (that is, without a human brain).

But could we split checks in different parts? What types of checks are there? M. Bolton describes questions that has to do with confirmation, verification, etc. Could we also break down checks into other categories such as checklists, test/quality patterns, guided checks and one-liner checks.

A checklist is a list of items that help you through a complex situation. The items in the checklist could be checks that need to be confirmed or verified to a certain extent.
A test/quality pattern is something that repeatedly gives suspicion that it might be broken so that you need to check it.
A guided check is what we previously called a scripted test. A set of steps that end with something that you wish to verify or confirm, something with a binary answer.
A one-liner check or a check idea is a brief statement of something that should be checked in a certain context that would give a binary answer, such as true/false, ok/nok or yes/no.

Checking clarifies an aspect of what testing is or is not, do these sub-categories help us clarify what we do with the checks? Could we find more types of checks?

Test Management Tools

Based on the assumption that there are at least two types of questions and at least two different types of answers, how do we structure and manage these today? The traditional management tools for testing can often handle all the above categories for checks, but they are unable to handle the answers from testing.

But what do we mean when we say handle in this case? Well, you can structure the checks, plan when you do them and report the result of them. You can then make reports on the overall progress and the overall quality of the system. You can also state what build, system or setup you used in the test. But since the tools cannot really handle the testing part, the progress and the picture on quality really becomes obsolete.

If we look at tools and techniques that are more aligned with testing such as SBTM, they are good at handling the testing story. They are not as good at handling checks, or rather the tools I have seen so far.

Can we and do we want to handle the result from our testing and checking in the same system? When working in small teams where there are fewer stakeholders to work for, the need for information sharing in a big system can be less important. But if you are working in an organization where you have teams in several countries and where the overall development organization can be thousands of people, it can be more important to share information in a bigger system. Still, are you actually able to share information to that many people in an efficient way just because you use an advanced management system? When a lot of people are involved in creating and sharing information there is also a bigger chance that the actual meaning is lost. You then oversimplify the situation that information sharing is easy. If the organisation is big, there is a big chance that the context changes over the organisation and that the information changes meaning over the organisation.

I’ve seen so many test management tools that try to solve the whole test process. By focusing on all aspects they also become quite crappy at all aspects. Picking the tools that are excellent at solving one problem can then be a lot more efficient even if you get to work with several tools. When working with test coverage, communicating your test ideas, reporting status, reporting progress and showing details on what you tested there is a big chance that one tool really cannot solve this for you. We know that we find new techniques from other disciplines that can solve a problem for us when testing. So why do many limit themselves to test case management systems?

Summary

Michael Bolton and many other great minds have explored and delved into this concept, I think we can find more pieces that we can shed some light on. I’ve identified a few areas and there are more to come.

References

[1] Elements of Testing and Checking – http://www.developsense.com/blog/2009/09/elements-of-testing-and-checking/
[2] The Checklist Manifesto – How to Get Things Right – http://www.amazon.com/The-Checklist-Manifesto-Things-Right/dp/0805091742