thoughts from the test eye

More than 3 years have passed since we published the first version of our Software Quality Characteristics. It is quite popular, and it is now translated to 8 languages by testing enthusiasts. But it’s about time to talk a bit more about how to use the list, where there are at least three typical scenarios:

Test Idea Triggers

This is the most common usage: read relevant parts of the list, and use it as inspiration for things to test in your product, or feature. Suitable when you have run out of ideas and need new inspiration. Don’t try to test all of them, because all of them won’t be important. But don’t discard any top category to easily either, the right testability suggestion might be your biggest time-savers.

It can be difficult to transform a generic description to actual test execution, so have a look at Lightweight Characteristics Testing for fast ways to get going. If you use the list many times, it is the right time to create your own customized list, with the appropriate characteristics for your situation; your own specific non-functional checklist.

(This is actually the origin of our poster; the print-out of Bach’s Quality Criteria became too cluttered with things we often wanted to test.)

Quality objectives/Non-functional requirements

The list can also be useful to understand what quality means for your software. This should probably involve other people than testers, so you get a good understanding from many perspectives. But the advice is to start without the list; define what quality means to you, and use your own words, because those will better describe what you want to accomplish. When you run out of ideas, then pick up the list of quality characteristics to see if you missed something relevant, or if you get ideas that can make your first ideas even better.

An important part is to try to make the quality statements really useful is to make them specific for the situation. “Easy to use” is very generic, so better examples are:

• First-time users should have no problems creating their first mind map
• Power users should very quickly be able to create complex mind maps (keyboard + mouse navigation)
• Product should adhere to 508 accessibility guidelines

As a tester, I don’t need these to be objectively measurable, they can still guide my testing effort, and help me focus and observe more broadly. Many other people want these to be easy to measure, and my guess is that’s why people don’t have time to inform about what we actually want to achieve.

Review of Test Strategy or Results

If you are in a situation where you should review a test strategy, or test results, the quality characteristics can be very handy to spot holes in the testing. Use the same thought process as above: “What does this mean to us, is it important?”

If you find that performance aspects are missing, ask if they just forgot to mention it, or if they should revise the strategy. Or even better, do this test on your own work, so you can improve immediately.