It is very difficult to describe the process of synthesizing test ideas. It involves a multitude of information sources, a sense of what’s important, and a dose of creativity to come up with ingenious test ideas, and effective ways to execute them.

The easiest way is to take the requirements document, re-phrase each item, and optionally add some details using equivalence partitioning.

The best way is to use a variety of information sources and generate testworthy test ideas that have a good chance of being effective.
An impossible way is to combine all important information in all possible ways.

Rather you should use each element that is important, and each combination you think is important.

Reviews are helpful, the actual product invites to some tests, and also the speed of the tests will guide you to the best ideas in your situation.

It is recommended to write down the test ideas, at least with a high-level granularity. If reviewing isn’t done, or the software already is available, it is faster to write them afterwards, together with the result.

Don’t try to cover everything, because you can’t. Rather make sure there is a breadth, and count on serendipity to find the information needed for a successful product. Some of the test ideas you will generate on-the-fly, during your test execution when you see what you actually can do with the software.

And don’t stop just because you reach an adequate test idea. Think some more, and you might find better ways, or new solutions to old problems.

I find no better way to elaborate further than to discuss different categories of test ideas:

Ongoing Test Ideas

Ongoing test ideas can’t be completed at once; they keep going as long as more relevant information is revealed. A good example is quality characteristics like stability; the more you test, the more you know about the stability of the product. You probably don’t do only ongoing tests (in the background) for important characteristics, but as a complement, it is very powerful, and very resource efficient.

Another example is ongoing usability testing for free, where the tester keeps the relevant attributes in the back of their head during any testing, and whenever a violation occurs, it is noticed, and communicated.

Classic Test Ideas

Classic test ideas deal with something specific, and can be written in “verify that…” format. They can be a mirror of requirements, or other things the testers know about.

For review and reporting’s sake, beware of granularity; rather use one test “verify appropriate handling of invalid input (empty, string, special ASCII, Unicode, long)” than a dozen of tests. In some situations you can even use “Verify explicit requirements are met”, and put focus on the more interesting test ideas.

Also take the opportunity to recognize which tests are better suited as automated tests, and which are safest to test automatically, and tool-aided and manually.

Combinatorial Test Ideas

Many bugs don’t happen in isolation (that’s why unit tests are very good, but far from a complete solution), therefore testing wants to use features, settings and data together, in different fashions and sequence.

Scenario testing is one way to go, pairwise is said to be good, and the tester with product knowledge can tell you at once which interoperability can’t be neglected.

Combinatorial test ideas use many test elements together, because we think they can have effect on each other, or we don’t know.

Unorthodox Test Ideas

Requirements are often written so they can be “tested” (but what usually is meant is “verified”.) This easily results in requirements being quantified and often in a contractual style, instead of describing the real needs and desires.

The risk of missing what is really important can be mitigated with testing that isn’t necessarily aimed towards falsifying hypothesis. “You are one of few who will examine the full product in detail before it is shipped [Testing Computer Software]

Unorthodox test ideas are open, based on something that is interesting, but without a way to put a Pass/Fail status, and more aimed towards investigation and exploration for useful information.

With creativity you can come up with original ways of solving a testing problem.
Usability testing can be addressed by asking any heavy user what they think.
Charisma might be best evaluated by having a quick look at a dozen of alternative designs.
If there are a hundred options, you might want to take a chance and only look at five random.

These are especially vital to get feedback on, since one unorthodox test idea can generate two others, that are better, or be dismissed as irrelevant.

Visual Test Ideas

Some things can’t be expressed with words. And many things are much more effective to communicate with images. Try to use visual representations of your tests, also because they can generate new thoughts and understandings.

Examples: state models, architectural images, technology representation, test elements relations, inspiring images of any kind

The visual test ideas, as all the others, can be transformed to other types, morphed to many, or what is best suited.

What and How?

Most test ideas will probably be focused on what to test. Some will also state how to test it. This can have many benefits, e.g. if you write you’re going to use Xenu’s Link Checker, and someone points out that there’s another tool that will fit the purpose a lot better.

These categories are artificial, and only used to better explain the processes.
When you know them, forget about them, and re-build your own model of the test design that suit your, your colleagues’ and your company’s thinking. Make sure that you change and add to your test ideas as you learn more.
Your questions, hypothesis and answers will emerge.

It is often said, with right, that you need to consider a lot more than the explicit requirements in order to be able to test a product well.
Often a few examples are included, e.g. something about the customer needs, or the necessity of reading the code, or knowing the technology the software operates in, or an understanding of quality characteristics in the specific context.
But I have not seen a long, yet brief, list of things that might be important as inspiration, so I thought we together could come up with a “checklist when building your own checklist”.
This is what I have so far:

MODELS

Requirements: explicit, combinatorial, implicit, incorrect, changing, vague, different, impossible, aiding
Specifications: conceptual, technical, functional, design, test specifications
Code: old, new, shaky, read, reviewed?
Help: online, pdf, web sites
Actual software: prototypes, in progress, released, competitors
Other people’s models

HISTORY

Previous versions: what did the old version do?
Bugs/Error catalogs: what bugs occurred for similar functionality?
Test Ideas/Cases/Strategies/Results: what can you learn from previous test efforts?
Claims: which features are used in marketing material?
Reviews: what has been said by others about your product?

USAGE

Support department: what experiences are channeled to support?
User scenarios: how many ways of actual usage of the software do you know of?
Customer stories: what problems are the customers trying to solve?
Dog food: can you use the product internally “for real”?
Competitors: software products, in-house tools, analog systems
Training material: learn about how customers learn your software
Business: needs, logic, information, knowledge, standards, laws

TECHNOLOGY

Environment: Hardware, OS, Applications, 3rd Party Components
Tools: development tools, (static) test tools, monitors, editors, brain
Systems: what does the big system look like?

PROJECT

Functionality: recently “improved”, core, problematic, high interoperability, complex, popular
Risks: important, omitted, forgotten, changing, unknown
Project plan: when, what, how?
Process: Agile/Waterfall-mix
Infrastructure: configuration management, test environment
Test execution context: what, when, where, why, who, how?
Quality objectives: what is always important?
Deliverables: executables, interfaces, all sorts of documentation, Release Notes, readme:s, meta data

PEOPLE

Team: developers, interaction designers, leaders, managers, technical writers, testers, experts
Stakeholders: have you talked to (product) managers lately?
YOU: your knowledge, experience and subjectivity
Users: needs, knowledge, feelings, impairments, personas, have you seen the real users in action?

SKILLS

Analytical thinking: model details and the big picture, follow paths
Critical thinking: see problems, risks and possibilities
Creativity: broaden test spectra, generate new ideas, lateral thinking
Factoring: break down to testable elements
Investigation: explore and learn
The Test Eye: wants to see errors, sees many types, looks at many places, looks often

SOFTWARE TESTING

Quality Characteristics: Capability, Reliability, Usability, Charisma, Security, Performance, IT-bility, Compatibility, Supportability, Testability, Maintainability, Portability, Localizability, Auditability
Generic test ideas: quicktests, tours, mnemonics, heuristics
Tricks: error-prone machine, Basic Configuration Matrix, attacks, Cheat Sheets
Information: books, courses, blogs, forums, web sites, conferences, conversations
Testing theory: with many different techniques/approaches you have a higher chance of finding important information

Suggestions for improvements are very welcome!!

Everything is not always relevant, but I recommend spending a few seconds on each sub-category. Think about what is important, and what would be worthwile to investigate more.
By using multiple information sources, including the actual product and customer needs, your tests will get a better connection with reality, they will be grounded.

For quite some time I have felt that the classic test design techniques don’t add up to the needs of software testing that tries to find most of the important information.
At EuroSTAR 2009 it dawned on me that it is time to describe the method that I, and many, many others, have been using for a long time.
By talking more around this, I hope we can spread the usage of this test design, improve the way we do it, and make it more understandable for other people, avoiding the comment: “oh, you’re error guessing.”
What I will describe could be called a method, activity or a skill, but I like to think of it as a test design technique, that is seen as a tester’s most powerful weapon, much better than equivalence partitioning, decision trees etc.

Grounded Test Design: learn a lot, and synthesize important test ideas.

To explain this further: Software testers should follow the thoughts of social science Grounded Theory; we should use many sources of information regarding the subject (requirements, specifications, prototypes, code, bugs, error catalogs, support information, customer stories, user expectations, technology, tools, models, systems, quality objectives, quality attributes, testing techniques, test ideas, conversations with people, risks, possibilities); we should group things together, use our creativity, and create a theory consisting of a number of ideas capturing what is important to test.
Grounded Test Design is for those that want to test a lot more than the requirements, that understand the need to combine things of different types, to look at the whole and the details at the same time.
This might seem heavy, but people have phenomenal capacity.
It has a lot more to it than the closest technique error guessing; for instance, it is not only about errors, it can also be investigations, or questions regarding dubious behavior.
It isn’t guessing, it is a qualified synthesis of knowledge.
And also: it isn’t (and doesn’t sound like) something done lightly, something unstructured, or unprofessional.

Being a technique, it is something that can be applied in many different situations. The most informal Grounded Test Design might happen extremely fast in the expert tester’s head when thinking of a completely new idea while performing exploratory testing; or it could be the base for test scripts/cases/scenarios, that are carefully investigated and designed in advance. Good software testing uses many different methods.
A formal Grounded Test Design (to be defined, tried and evaluated…) might be best used by someone completely new to a product, or maybe most efficiently performed in diversified pairs.

Grounded Test Design has a loose scientific, theoretical base in Grounded Theory (Strauss/Corbin), used in social science as a qualitative, thorough analysis of a phenomenon.
The scientists examine the subject very carefully, gather a lot of material, document codes and concepts, combine them to categories, from which a theory is constructed.
There is no hypothesis to start with, as in the traditional natural science, and that’s quite similar to software testing; it would be dangerous if we thought we knew the important questions at once.
I think it is good to be slightly associated with this theory, especially since it emphasizes that the researcher must use her creativity.

On the other hand; do we really need this word, maybe we already have enough words about how to do good software testing?
Or is it a good word to have in reserve when explaining to managers why testing is difficult, takes a lot of time, and must include manual work?
Or should we go further with this and try to develop a framework for performing structured analysis of everything that is important for testing?

Opinions are very welcome.