thoughts from the test eye

Our minds are fantastic, but can be biased and make dubious decisions. As I read Kahneman’s Thinking, Fast and Slow, I thought about test strategy and found some interesting stuff, based on mistakes I have seen, and done.

Answering An Easier Question

“How could we possibly test everything important?” is a very difficult question. It is natural, but not necessarily good, to exchange with a question that is easier to answer:

• How should we test the new functionality?
• How should we do the same testing as last time, but faster?
• What can we automate?
• How long does the test plan need to be?

When we answer the simpler question, we subconsciously think we have answered the difficult question…

This can also happen when we are asked to estimate how long something will take to test, before we have considered which strategies that are appropriate. Later on, we will probably choose the strategy that will give the result from the estimation. Another version is to split up the problem in smaller pieces, e.g. test phases, where responsibility is distributed, resulting in loss of the whole picture, and important areas uncovered.

Cure: Seek the difficult questions. For each part of your test strategy, consider if the answer (or the question!) is too narrow. Especially look for opportunities where a specific strategy can be used for more additional testing missions.

Example: When performing manual testing, change platforms and configurations as you go, so you won’t need a special compatibility phase. When appropriate, do spot tests for additional platforms.

What You See Is All There Is (WYSIATI)

It happens that the strategy only consider the testers’ (and developers’) testing, when other strategies might be more powerful. But also the opposite happens, that customers are doing acceptance testing doesn’t automatically mean that no one else should consider if the product is “good”. A common mistake is to only “see” the requirements document, and believe those are the only things that are important to test.

Cure: Discuss what else to test, not only what is inside the project plan.

Halo effect

If something is good or bad, we tend to transfer those attributes to other, unknown, areas. As an example, a long and handsome test plan can make us believe that the strategy is reasonable; and a sloppy bug report make us doubt the content. A negative effect happens when the behavior of the product on one area effects our expectations on other things. When something looks good, we might base our strategy on the belief that the rest also is good; if we see a crash at once, we might think that further testing is pointless.

Cure: Don’t build important decisions on single observations; look more, don’t have prejudices about good or bad product quality (let the observations judge.

Counter-example: Distribute your strategy in plain notepad format, so the content is in focus.

Illusion of validity

You run the same strategy as last time, because you found important bugs. That equally many were missed, and that it was very time consuming, is disregarded. One example can seem to make a strategy valid, 1 important issue after investigating 100 backward compatibility files make it worth the effort (which might be true, but there might be other ways that would be better.)

Cure: Even experts can’t predict the future, so make sure to have diversity in your test strategy.

Example: Since we have run free exploratory testing for our final regression tests for the last releases, we will this time do it function by function.

Optimistic bias

Our plans tend to be optimistic; we think all days will be good, downhill with the sun and wind supporting us. This also happens to our testing strategies, they aim higher than we can reach, especially since the unexpected always happens. That’s why it is important to communicate which parts of the strategy will receive most time, and which parts will be tested lightweight (it is seldom wise to skip relevant parts totally.) Your strategy should be realistic, and consider excessive optimism and unknown unknowns.

Cure: Make sure you communicate your priorities, so the less important parts can be skipped, or tested shallowly.

Focusing illusion

“No separate part of the testing strategy is as important as you imagine when you think about it.” I still get trapped by this one, especially when I try to convince someone about a certain way to test. The truth is that a lot of test coverage for the most important stuff is overlapping; a severe installation problem is captured by any test method, an alert manual tester can easily see usability, performance and security problems, and the specialists for each area can also look broadly.

Cure: De-focus. Look at the whole, discuss with people not obsessed with you current area.

Regret and responsibility

An important explanation for dubious test strategies could be fear of regret and blame. If you run the same strategy as last time it is less risk for complaints than if you actively changed the way you tested. If you can reference a best practice, people might think that reasonable choices were made. But as with other biases, one should rather choose what one thinks is best, than keeping your back free.

Cure: Diversity in the test strategy, and courage to change when you see the results.

Example: An automation strategy hasn’t produced much in one year. Add more resources or start all over?

Finale

Bias is a natural people-thing that often helps us. It can’t be avoided, but it can be managed (by being aware of it.)

In his book, Kahneman often mentions luck, which is important also for testing. But good testers are often lucky, and a good test strategy creates opportunities for good fortune. In a sampling business, serendipity is nothing to be ashamed of!