When I wrote blog post The Complete List of Testing Inspiration, I didn’t think so much about many testing efforts being totally based on requirements and specifications.
I took for granted that we know that requirements are incomplete and wrong, and that we should learn from many places.
But when reading a good book such as Lee Copeland’s A Practitioner’s Guide to Software Test Design you see that the classic black box testing procedure is to base your test design solely on requirements and specifications.
It’s the same as with code coverage models, the most interesting stuff is what isn’t there.

Maybe we need to sharpen the argumentation for this, so here are my first attempts:

1. What problem is testing trying to solve?
I think there are more situations like this:
“We know that there will be defects in the product we are building. We need your help to make sure that the released product can be used by customers without problems, and satisfactorily help them with their tasks.”
than like this:
“We want to make sure that the actual product meets the statements in the requirements document.”

2. Requirements checklist of one
Did an omnipotent write the requirements?
(We know more when testing than when requirements are written; testers look at other details; testers also test the requirements, how things turned out.)

3. The intuitive argument
A visual representation, like the software potato: an image showing that requirements cover less than what’s important, that is less then everything.

4. Requirement realism
Requirements aren’t explored, but if all were written according to Exploring Requirements by Gause and Weinberg, we would probably be in a better position. We would not only know what needs to be in the product, we would also know about Preferences, Constraints, and “Want it without Cost”

One argument should suffice, but which one, and how can it be polished?

Exploring Requirements: Quality Before Design is an excellent book written by Donald C. Gause and Gerald M. Weinberg.
It is primarily about requirements, but it is an excellent read for everyone involved in doing something that hasn’t been done before.

As a software tester, it highlights, and helps, my own problems with understanding all important aspects of what the product is good for.
I wish all requirements were written according to this book; it would make testing a lot easier, especially when trying to understand attributes, preferences, expectations and motivations.

They describe a human-centered requirements process, both regarding the people collaborating, and the product, that will be used by people, that have feelings.
At one moment I felt that it was too much measurements, but on the next page I read:
However, keep in mind what von Neumann said, “There’s no sense being precise about something if you don’t know what you’re talking about”, and don’t get bogged down in metrics.

Of specific interest to testers is the notion of Frill/”Get It If You Can”: functions (or attributes) that are desirable, but shouldn’t cost anything.
My testing interest is two-fold: first, even if these are ignored, you should watch out for violations to these, because they are (implicit) bugs.
Second, manual system testers are often good at finding out how nifty small additions would be, so testers can help raise priority of frills, or create new and better ones.

I also want to mention the User Satisfaction Test, which looks very useful, maybe even for testing status reporting??

So, with a title like this, does the book contain any hints on documenting tests in advance or not?
Yes, it does: “To be most effective, black box test construction must be done before you start designing solutions.”
1-0 to documenting test ideas early on.
Page 258 (original 1989 edition) also describes the very first test design technique:
asking ‘What if’ questions
Page 94-103 deals with the very first test analysis technique:
“Mary had a little lamb” heuristic

There’s a lot more that can be used in any situation, e.g. Context-Free Questions, and Rule of Three: “If you can’t think of three things that might cause your great idea to fail, all that means is that you haven’t thought enough about it yet.”

The book is fluently written, and contains a lot of entertaining and illuminating stories, and a beautiful ending.
Endings like that are not necessary when merely writing a blog post.

It is good to strive for better requirements by critical analysis (and looking for what’s missing), but there is a danger in complaining about untestable requirements.
If those vague requirements are changed (made too specific) or removed, the words in the requirements document have less meaning, and less chance of guiding towards great software.

And there are no such things as untestable requirements. There are requirements that can’t be verified, that can’t get a true/false stamp, but you can definitely test the software and look for things that don’t match the essence of the vagueness.

An example: “the feature should be easy to operate” is difficult to prove right or wrong, but very easy to evaluate subjectively after doing some manual testing.
If the requirement is changed to “minimum no. of mouse-clicks to perform common operations”, you might catch some issues, but some other, more important things, might be lost in translation.
And if the requirements are split into many, many smaller pieces, you might lose less information, but end up with a too complex document that is very expensive to create and maintain.
It’s not a bad thing to be specific, but that’s not feasible for everything.

There’s an underlying assumption I should tell you about:
I do not think requirements should be contractual, they should rather be aiding – they should help the development team produce good software.
Since requirements neither can be complete nor perfect, we should rather take advantage of oppurtunities that arise, and create something that can solve problems. If the essence of the unspoken requirements are captured, it might not matter that a few specifics aren’t met.

Testers should keep in mind that there’s a greater whole we’re aiming for, and do our best with what we have, so be it unverifiable requirements.

Software Potato

 The square symbolizes the features and bugs you will find with test cases stemming from requirements (that can’t and shouldn’t be complete.)
The blue area is all bugs, including things that maybe no customers would consider annoying.
The brown area is all important bugs, those bugs you’d want to find and fix.

So how do you go from the requirements to “all important bugs”?
You won’t have time to create test scripts for “everything”.
So maybe you do exploratory testing (thin lines in many directions), and hope for the best.
Or maybe you test around one-liners (thicker horizontal lines), that are more distinct, that are reviewed, and have a better chance of finding what’s important.
Either option, some part luck, and a large portion of hard work is needed.
But I think you have a much better chance if you are using one-liners, especially if it’s a larger project.

Later I have realized that one-liners aren’t essential; that this problem has been solved many times at many places with many different approaches.
What is common could be that testers learn a lot of things from many different sources, combine things, think critically and design tests (in advance or on-the-fly) that will cover the important areas.
Maybe we need a name for this method; it could be Grounded Test Design.

Statements in specifications try to clarify and are inevitably an interpretation of what the author thinks need to be more specific. I.e., they try to be a more specific model than what existed before the spec. And “Essentially, all models are wrong, but some are useful” (http://en.wikiquote.org/wiki/George_E._P._Box).

Every specification you encounter is persons’ interpretations, and  not necessarily true.

This means that you as an inquisitive tester have a lot to do by questioning the specifications. The questioning will help you to form a model of the software that is better than if you only had read and accepted the spec as it was.

Specifications cannot be complete and especially regarding things that the program shouldn’t do. It is probably not stated that the software shouldn’t use too much memory or processor for certain operations; it is not stated that the screen shouldn’t flicker, or that all text should be easy to read with all different font settings. Other typical omissions are interactions with other systems; things you expect from all applications under that operating system, internet browser, connected software etc.

You cannot expect a specification to be complete, in most (all? many?) cases, the thing produced by the specification is more important than the document about it. The hardest challenge for the inquisitive tester is to question a lot, but only for those things that are important.

——————–

Who will use the specification? What will they use it for? Will it meet their requirements?

What is it all about? Really?

What areas are left out?

Who is the writer? Does he/she usually miss certain things?

Are there many writers? Does this make the whole less tangible?

Are there many reviewers? Are they using different perspectives?

Is the writer vague, insecure and confusing about certain areas?

Is the specification consistent?

Is the specification consistent with other related specifications?

Is the specification consistent with other different features and combinations of those?

Are all functional and non-functional requirements covered?

Are there dubious thoughts about the wished for functionality?

Are there other sources of information that can be useful?

How is the style of the language affecting the specification?

What quality attributes are the most important, e.g. how is Security weighed against Performance and Usability?

Does it match the system requirements?

Does the specification focus on what is most important?

Does the specification reflect the model of what you think is described?

Are there any new terminology? Will this affect other documentation such as help files?

Is the new terminology consistent with other specifications?

What does the Internet say about the newly chosen terminology? Will there be any misunderstandings?

—————–

If there was no specification, could it be described in a completely different way?

I like to use categorizations to structure my understanding of a subject; and after the simplifications are made and I think I understand it well; the structures can be ripped apart, and you get a bit less confused by the complexity of reality.

There are many forms of requirements, these are some a tester should look out for:

Explicit Requirements
These are the requirements found in the requirement documents. You are probably using them in your testing, making sure that they match the functionality.
This is a quite small part of software testing as I see it.

Implicit Requirements
These are requirements that can be found by combining different requirements that are intertwined.
They could originate from general statements like the program should never crash, or the program should be easy to use, which has implications for many other requirements.
They could also become very large, e.g. support all possible input, or support Python scripting.
They are an effect of vague requirements, and they are a natural part of software development; it would be insane to document everything in advance. Tester can deal with it and understand what is important.

Unspoken Requirements
These are things that many users expect from a program, but they are seldom listed in the requirements document.
Typical examples are behave in same way as other applications on this platform, not leave any garbage files after running, or be appealing to most users.

Incorrect Requirements
The writers of the requirement documents don’t know everything in the world, sometimes they are wrong.
There can be small errors, e.g. inconsistencies between requirements, and huge mistakes, because they didn’t understand the user’s true needs.

Changing Requirements
Sometimes requirements need to be changed, which is something that testers shouldn’t object (too much). The requirements are most likely changed in order to make a better product, and that’s what we all are working for. But when they are changed, or added at a late stage, it can be difficult to challenge them, and test them really good, simply because you are under time-pressure.
We can’t do more than our best, but that’s often enough.

Vague Requirements
I used to dislike vague requirements that were very difficult, almost impossible, to test, but now I think they are good to have.
Not that you should be vague on purpose, but quality attributes like usability, performance etc. can never be detailed and capture the important thing: that customers will be more than satisfied.
It gives you a challenge as a tester; you need to use your feelings and imagination to come up with test ideas, and results that point to a positive or negative indication of result. You can’t hide between some numbers, and must stand for if you think the requirement is met or not.

Hype Requirements
These can be difficult to handle. Often they come in the shape of specifying too much detail, e.g. save settings in an XML file, just because XML is hype (this was 10 years ago, so replace with SOA or the cloud or the hype your company believes in, right now.)
They might be out-of-place, put there in order to be allowed to start the project, but they can also be important, exploiting the hype, or just being a perfect match for this specific application.
As a tester, it’s often not much more to do than accept the hyped requirements, especially if it is accepted (or initiated) by the developers.
But probably you need to learn more about the hype, often there are (at least some) good things inside them.

And regardless of how well all these categories of requirements are implemented and tested, will the application be really, really, super good?