Exploratory Testing Best Practices* Rikard Edgren
When testing software, the very best inspiration and reality check comes from the software itself. It helps you test beyond requirements, and investigate what the software really is capable, and incapable, of.
These are my best practices for exploratory testing.
1. understand what’s important – we can’t test everything, and we can’t find all bugs. But it can be feasible to find all important bugs. This is very difficult, doesn’t involve a great deal of luck, but a good understanding of multiple, and diverse information sources. An Exploratory Testing trap is to spend a lot of time finding bugs that are interesting to the tester, but not to the project.
A key to important matters is to be subjective. I guarantee you, if your software is made for people, they will be subjective when they are using the product. Use your humanity and your feelings, they are worth more than the explicit requirements. The truth is the subjectivity (Kierkegaard)
2. be open to serendipity – the most important things are often found when looking for something else. If we knew exactly where all bugs were located, it would suffice with automation or detailed test scripts (or maybe we wouldn’t have to bother with tests at all?) So make sure that you look at the whole screen, and many other places as well. Odd behavior might be irrelevant, or a clue to very important information. Testing always involves sampling, and serendipity can be your friend and rescue.
3. work hard, but don’t – manual, exploratory testing can give enormous coverage in a short time when pieces are in the right place. At the same time, you should pause your fast runs and look from different perspectives, both in order to see more things, but also to not get too tired. Sometimes you need to investigate details for a long time without progress, sometimes you need to do something completely different to get fresh eyes. Stay focused, lose focus. Exploratory Testing Dynamics is a profound document with lots on this.
Structure and discipline are key parts of any successful testing effort, but also to combine disparate information sources (often called creativity.)
4. work close with developers – in physical location, and in time (tight feedback loop: code-test-fix-verify). Talk to them instead of reporting a vague bug; earn their trust, and share information.
But testers should think differently, if everybody looked at the software in the same way, we wouldn’t get the broad coverage it (hopefully) will be exposed to after release.
It is good to work with other roles as well, but developer collaboration is the most important.
5. one-liner test ideas – if you lightly document the test ideas you intend to execute, you can give ideas to developers even before they have written the code (with the inevitable bugs.) You can also get feedback on importance and improvements by fellow testers, other interested and stakeholders. With appropriate granularity you can go below 100 test ideas, and make it possible for others to understand the testing that will be performed, and comment on what is missing.
The list could be made longer (broad learning, work in other roles, CRUSCPIC STMPLA, see the big picture, SBTM, creativity…), but I think it’s better to focus on a few important. Who knows, I might be totally wrong, and don’t want to spend too much of your precious time.
–
* There is no such thing as something absolutely, objectively best. This goes for all situations.
An Olympic champion was the best at that moment according to the rules that were used in that competition.
Newton’s physics were the best for a while, and are still good for approximations in real life.
“The Beatles are the best band in history” is a subjective statement, shared by many.
So in this article, feel free to exchange “Best Practice” with “good examples for me that I hope are inspiring”.
Ask Wittgenstein, language is a game. You decide what to do with the words.
A very good list Rikard!
When reading the list I realize that it resembles mine (imaginary one).
I have been sloppy with the one-liners, but the diversity of the projects I have worked in lately have differed so much in the documentation needs that there haven’t been any “best practice” to apply…
I would like to add to my list:
“Know who the stakeholders are and their needs”. Especially important when working in diverse projects. It helps me to focus on what matters for the stakeholders. And it also gives you information on how you should report status and results. While this resembles your first point to some degree, it might better be a bullet of its own.
Something else I would like to add to my list:
“Hook up with someone and do some pair-testing now and then.” In order to learn more about testing and testing techniques it is always fruitful to pair up with someone and learn from that person. And the test ideas that can come from two collaborative minds is sometimes unbeatable. Sadly, this does not happen too often for me nowadays, but when it happens it is bliss.
nice additions!
I would put “stakeholders” as a sub-part of what’s important.
Pair-testing could be part of two different, generalized ideas: “collaboration” and “learn/get new perspectives”.
Another thing about one-liner test ideas is that they can be used in your session notes. Instead of only documenting the bugs, or document most of what you do, it can be appropriate to abstract your actions into one-liners that describe the testing you have performed.
Another thing that comes to mind: if these Exploratory Testing Best Practices are applied to scripted testing, are you then doing exploratory testing?
Regarding your last question.
Yes. And No.
Sometimes you are allowed to apply these in a scripted testing environment and sometimes you aren’t.
I have experienced at least two projects where any of the items on the list would be “illegal” and not accepted. I.e. if you see it from a testers perspective. Maybe one or two may have been applicable for the test designer in those projects.
Even if you are thinking in those projects it is not rewarded and, to be honest, not necessary.
I do not like this, but this happens. I do try to break this behavior whenever I see it, but it is also a matter of where in the organization you are in order to be able to change it.
Nice list! Not due to completeness or fresh ideas but due to clarity. Very to-the-point and easy to understand.
I would say that it is very much possible that you are doing exploratory testing applying these points to scripted testing.
Take the first point as an example; You select test cases to run based on what you learn is more important or you might as a result of your first run learn that some areas seem to be less complete, or more error prone, that others. Then you can actively select the scripted test cases that cover that area. Adapt your test execution based on what you learn as you go along.
Thanks, Oscar!
It would have been too mind-gobbling with a brand new idea being a best practice, right? (I guess you didn’t mean the ideas were mouldy?)
I agree that the list isn’t complete, and I have a feeling that you have a practice that qualifies on the list.
Can you share?
You’re right, that’s what I meant.
I would add something in the line of “Be critical against you’re own work too”.
What I mean by that is that it is important to look at your own methods and habits of working. To be aware of where improvements can be made or even when and why focus (or defocus) may have been disturbed. Sometimes your minds wanders or you settle in to some way of doing things
Pair testing is probably a good tool to do that though, if you can muster up a partner!
Oscar, are you suggesting that testers can make mistakes too?
Good suggestion, and I was about to object that your suggestion is too general, and applicable to almost everything, not just testing.
I realized that this goes for my list as well (and Henrik’s addition), except for the one-liners.
Should a Best Practice* be specific for the area it describes?
@Rikard, isn’t it so that your list becomes specific by having a title including the area of concern: “Exploratory Testing”? That way, the practices gets a context in which you say that they are suitable in/for.
I mean, “one-liner test ideas” is the only thing on the list specific on “test”, but it is a kind of a checklist that in this case contains ideas on what to test. So it is a test-specific checklist.
It does not matter if your practices are applicable to almost everything, what matters is that you have singled out 5 that you think are important for Exploratory Testing. There must be thousands of practices that aren’t hitting the top 5 list for this context. E.g. “Sit comfortably”, “Follow the process exactly”, “Wear gloves”, “Eat vegetarian food”, “Place the plate exactly in the middle of the micro wave oven”, “The specification must be approved”, etc.
Good points, Henrik.
And I liked your comparison practices, they were fnu.
“Wear gloves” will defeinitely be on my top 10 list!