thoughts from the test eye

Introduction

In 2009 Michael Bolton had a talk at Agile 2009 from which he later on wrote about a distinction between testing and checking [1]. He has since then elaborated more in the area and digged deeper into it [2], and continue to do so [3]. By clarifying what we are doing at a certain time it makes it easier to determine what tools and heuristics to use among other things. By making this distinction I think it helps us as testers to do a better job and to better be able to explain to other non-testers why we need to do both testing and checking, but also what the difference is.

The idea that you can (and should) automate all testing is not interesting. It is like discussing automation of all decision making. In a discussion like this we can instead offer that checks are usually good to automate, but testing is not as easy since there is no true pass/fail result when it comes to open-ended information, that might have different value to different persons. James Bach makes an example how this is handled in BDD [4] or rather making it obvious that some checks might be easy to confirm while others are close to impossible.

When I talk about testing vs. checking and touch the subject about sapience [5], there are many who are known to promote the checking part as their main activity as a tester, who then feel uncomfortable. What is really important is to point out that they do exploratory testing as well [6], but perhaps not framing [7] or structuring it well. I’ve heard arguments that exploratory testing is less structured and formal, well perhaps for you it is. Why not change that?

Simon Morley makes some interesting observations in his post on Sapient checking [8] and I agree with him that it might confuse more if we talk about testing and checking when we are delivering information to inform decisions. Should the focus in our information be on if it was checking or testing being made? For most cases, I believe we should not.

Test Driven Checking

If we assume that testing and checking is intertwined, you might still consider that in some cases one thing drives the other or that at least one activity is in focus while the other is in the background.

Depending on how well I have organised my charters and missions for testing during a project I test differently. Sometimes when I run a test session (following principles from SBTM [9])  I do note taking on what tests and checks I have on the system, mixing the two activities. Afterwards I might go through a checklist and mark items off as done. This might be coverage related, requirements related or whatever was suitable to have in a checklist. I might have stored most of the information in my session notes, but I might still go into other areas to mark it there as well, if that was easily done. A college did exploratory testing, without note taking, and then went into an excel sheet to check off coverage areas. He then reported progress and coverage based on that.

To avoid missing some checks you might consider that after you have tested in an area and you are ready to move on, you take a peek at things to check in that area. By doing so you might avoid the confirmation bias [10] and unintentional blindness. You might also get new ideas for testing when seeing things that you should check.

When you do regression testing you might consider to first test around the area of the bug, then as a final step check if the reported bug is in fact fixed.

Check Driven Testing

This way of testing is probably the most common one by those who are unfamiliar with concepts of exploratory testing. As I see it, you might have a list of checks (or for some scripted tests) that you check, but you might test around them as well. In this case the checks are driving the testing. When you have one-liner test ideas as your main focus, I think it is most common that you let them drive you. So depending on how you define them you will have different focus. Simplifying it, are they “What if”-focused or “Verify that”-focused?

When you retest a fixed bug you might select the methd to check that the bug as it was reported is actually fixed, then you continue with what valuable testing around to see that nothing else has broken in the vicinity, as far as you know.

If you do a testing tour you might set out some designated targets (check points) and along the way do your testing reporting what you actually find out.

If you consider how kids might act when they walk, on their own, on one side of the street in one direction and all of a sudden they see their parents on the other side. The kid forgets everything they had learnt about traffic rules and risks, then just runs to the parents by crossing the street. I believe we can see this with testing and checks as well. When you get to the point where you can do the check, you forget all other things that you should really be testing, thus a case of unintentional blindness.

Conclusion

This is just one way of looking at it, this is no real conclusion. For me it helps to make a distinction between doing Test Driven Checking and Check Driven Testing. By being aware of the distinction you might plan your testing differently, you might consider a bit more what where you put your checks and where you do your testing. They will affect each other and they will be drawn to each other.