Kahneman and Test Strategy Bias Rikard Edgren

Our minds are fantastic, but can be biased and make dubious decisions. As I read Kahneman’s Thinking, Fast and Slow, I thought about test strategy and found some interesting stuff, based on mistakes I have seen, and done.

Answering An Easier Question

“How could we possibly test everything important?” is a very difficult question. It is natural, but not necessarily good, to exchange with a question that is easier to answer:

  • How should we test the new functionality?
  • How should we do the same testing as last time, but faster?
  • What can we automate?
  • How long does the test plan need to be?

When we answer the simpler question, we subconsciously think we have answered the difficult question…

This can also happen when we are asked to estimate how long something will take to test, before we have considered which strategies that are appropriate. Later on, we will probably choose the strategy that will give the result from the estimation. Another version is to split up the problem in smaller pieces, e.g. test phases, where responsibility is distributed, resulting in loss of the whole picture, and important areas uncovered.

Cure: Seek the difficult questions. For each part of your test strategy, consider if the answer (or the question!) is too narrow. Especially look for opportunities where a specific strategy can be used for more additional testing missions.

Example: When performing manual testing, change platforms and configurations as you go, so you won’t need a special compatibility phase. When appropriate, do spot tests for additional platforms.

What You See Is All There Is (WYSIATI)

It happens that the strategy only consider the testers’ (and developers’) testing, when other strategies might be more powerful. But also the opposite happens, that customers are doing acceptance testing doesn’t automatically mean that no one else should consider if the product is “good”. A common mistake is to only “see” the requirements document, and believe those are the only things that are important to test.

Cure: Discuss what else to test, not only what is inside the project plan.

Halo effect

If something is good or bad, we tend to transfer those attributes to other, unknown, areas. As an example, a long and handsome test plan can make us believe that the strategy is reasonable; and a sloppy bug report make us doubt the content. A negative effect happens when the behavior of the product on one area effects our expectations on other things. When something looks good, we might base our strategy on the belief that the rest also is good; if we see a crash at once, we might think that further testing is pointless.

Cure: Don’t build important decisions on single observations; look more, don’t have prejudices about good or bad product quality (let the observations judge.

Counter-example: Distribute your strategy in plain notepad format, so the content is in focus.

Illusion of validity

You run the same strategy as last time, because you found important bugs. That equally many were missed, and that it was very time consuming, is disregarded. One example can seem to make a strategy valid, 1 important issue after investigating 100 backward compatibility files make it worth the effort (which might be true, but there might be other ways that would be better.)

Cure: Even experts can’t predict the future, so make sure to have diversity in your test strategy.

Example: Since we have run free exploratory testing for our final regression tests for the last releases, we will this time do it function by function.

Optimistic bias

Our plans tend to be optimistic; we think all days will be good, downhill with the sun and wind supporting us. This also happens to our testing strategies, they aim higher than we can reach, especially since the unexpected always happens. That’s why it is important to communicate which parts of the strategy will receive most time, and which parts will be tested lightweight (it is seldom wise to skip relevant parts totally.) Your strategy should be realistic, and consider excessive optimism and unknown unknowns.

Cure: Make sure you communicate your priorities, so the less important parts can be skipped, or tested shallowly.

Focusing illusion

“No separate part of the testing strategy is as important as you imagine when you think about it.” I still get trapped by this one, especially when I try to convince someone about a certain way to test. The truth is that a lot of test coverage for the most important stuff is overlapping; a severe installation problem is captured by any test method, an alert manual tester can easily see usability, performance and security problems, and the specialists for each area can also look broadly.

Cure: De-focus. Look at the whole, discuss with people not obsessed with you current area.

Regret and responsibility

An important explanation for dubious test strategies could be fear of regret and blame. If you run the same strategy as last time it is less risk for complaints than if you actively changed the way you tested. If you can reference a best practice, people might think that reasonable choices were made. But as with other biases, one should rather choose what one thinks is best, than keeping your back free.

Cure: Diversity in the test strategy, and courage to change when you see the results.

Example: An automation strategy hasn’t produced much in one year. Add more resources or start all over?


Bias is a natural people-thing that often helps us. It can’t be avoided, but it can be managed (by being aware of it.)

In his book, Kahneman often mentions luck, which is important also for testing. But good testers are often lucky, and a good test strategy creates opportunities for good fortune. In a sampling business, serendipity is nothing to be ashamed of!

David Högberg October 8th, 2013

..and the link to this post goes directly soaring into the twitterfeed! Thanks for sharing your thoughts!

Rikard Edgren October 8th, 2013

I hope you share any new ideas also as a comment here!

David Högberg October 10th, 2013

I thought you’d never ask!

All jokes set aside, I’ve been reluctant to leave my thoughts in print here since I am not sure I can add anything of value to what you have written.
“A man’s got to know his limitations.” Dirty Harry

But here goes.

The expression “test strategy” is often uttered together with the verb “write”. It is something you write down. Why do you write it down? Who will read it?

Very often during the actual testing you will encounter unexpected situations, strange circumstances and learning nobody had imagined. All these things may alter and shape the testing and the strategy as time goes by. Will the written test strategy have any value after a couple of weeks? I am guessing here it is unlikely that anyone will bother to update the strategy document.
If I suddenly die and another tester takes my place, will all that detailed and practical stuff I wrote down in a strategy help the new tester to get the hang of the current testing process? If the answer is yes, then the the test strategy was a good thing to both write down and to save.

So what I am trying to say is that during that period of time when you are actually writing down the test strategy, the creative process of producing the strategy might in itself help you and those around you to get a better understanding about what´s going to get done. But when the writing is done and the document has been read and approved, has it any value left?

I want my test strategy to be resilient over time and be to able to act as a clarifying
piece of information irrelevant to when it is being read or by whom.

Rikard Edgren October 10th, 2013

There are two test strategies, the documented, and the mental.
They are not the same…
I find it useful to communicate both in written format and in conversations.

The most important value usually comes from creating an communicating it, but for newcomers along the way, it should help.
And to read them months/years after can also be very interesting (or needed for auditing purposes.)

Resilient might be good, but watch out for the risk of being too general.