Synthesizing Test Ideas Rikard Edgren
It is very difficult to describe the process of synthesizing test ideas. It involves a multitude of information sources, a sense of what’s important, and a dose of creativity to come up with ingenious test ideas, and effective ways to execute them.
The easiest way is to take the requirements document, re-phrase each item, and optionally add some details using equivalence partitioning.
The best way is to use a variety of information sources and generate testworthy test ideas that have a good chance of being effective.
An impossible way is to combine all important information in all possible ways.
Rather you should use each element that is important, and each combination you think is important.
Reviews are helpful, the actual product invites to some tests, and also the speed of the tests will guide you to the best ideas in your situation.
It is recommended to write down the test ideas, at least with a high-level granularity. If reviewing isn’t done, or the software already is available, it is faster to write them afterwards, together with the result.
Don’t try to cover everything, because you can’t. Rather make sure there is a breadth, and count on serendipity to find the information needed for a successful product. Some of the test ideas you will generate on-the-fly, during your test execution when you see what you actually can do with the software.
And don’t stop just because you reach an adequate test idea. Think some more, and you might find better ways, or new solutions to old problems.
I find no better way to elaborate further than to discuss different categories of test ideas:
Ongoing Test Ideas
Ongoing test ideas can’t be completed at once; they keep going as long as more relevant information is revealed. A good example is quality characteristics like stability; the more you test, the more you know about the stability of the product. You probably don’t do only ongoing tests (in the background) for important characteristics, but as a complement, it is very powerful, and very resource efficient.
Another example is ongoing usability testing for free, where the tester keeps the relevant attributes in the back of their head during any testing, and whenever a violation occurs, it is noticed, and communicated.
Classic Test Ideas
Classic test ideas deal with something specific, and can be written in “verify that…” format. They can be a mirror of requirements, or other things the testers know about.
For review and reporting’s sake, beware of granularity; rather use one test “verify appropriate handling of invalid input (empty, string, special ASCII, Unicode, long)” than a dozen of tests. In some situations you can even use “Verify explicit requirements are met”, and put focus on the more interesting test ideas.
Also take the opportunity to recognize which tests are better suited as automated tests, and which are safest to test automatically, and tool-aided and manually.
Combinatorial Test Ideas
Many bugs don’t happen in isolation (that’s why unit tests are very good, but far from a complete solution), therefore testing wants to use features, settings and data together, in different fashions and sequence.
Scenario testing is one way to go, pairwise is said to be good, and the tester with product knowledge can tell you at once which interoperability can’t be neglected.
Combinatorial test ideas use many test elements together, because we think they can have effect on each other, or we don’t know.
Unorthodox Test Ideas
Requirements are often written so they can be “tested” (but what usually is meant is “verified”.) This easily results in requirements being quantified and often in a contractual style, instead of describing the real needs and desires.
The risk of missing what is really important can be mitigated with testing that isn’t necessarily aimed towards falsifying hypothesis. “You are one of few who will examine the full product in detail before it is shipped [Testing Computer Software]
Unorthodox test ideas are open, based on something that is interesting, but without a way to put a Pass/Fail status, and more aimed towards investigation and exploration for useful information.
With creativity you can come up with original ways of solving a testing problem.
Usability testing can be addressed by asking any heavy user what they think.
Charisma might be best evaluated by having a quick look at a dozen of alternative designs.
If there are a hundred options, you might want to take a chance and only look at five random.
These are especially vital to get feedback on, since one unorthodox test idea can generate two others, that are better, or be dismissed as irrelevant.
Visual Test Ideas
Some things can’t be expressed with words. And many things are much more effective to communicate with images. Try to use visual representations of your tests, also because they can generate new thoughts and understandings.
Examples: state models, architectural images, technology representation, test elements relations, inspiring images of any kind
The visual test ideas, as all the others, can be transformed to other types, morphed to many, or what is best suited.
What and How?
Most test ideas will probably be focused on what to test. Some will also state how to test it. This can have many benefits, e.g. if you write you’re going to use Xenu’s Link Checker, and someone points out that there’s another tool that will fit the purpose a lot better.
These categories are artificial, and only used to better explain the processes.
When you know them, forget about them, and re-build your own model of the test design that suit your, your colleagues’ and your company’s thinking. Make sure that you change and add to your test ideas as you learn more.
Your questions, hypothesis and answers will emerge.