Statements in specifications try to clarify and are inevitably an interpretation of what the author thinks need to be more specific. I.e., they try to be a more specific model than what existed before the spec. And “Essentially, all models are wrong, but some are useful” (http://en.wikiquote.org/wiki/George_E._P._Box).

Every specification you encounter is persons’ interpretations, and  not necessarily true.

This means that you as an inquisitive tester have a lot to do by questioning the specifications. The questioning will help you to form a model of the software that is better than if you only had read and accepted the spec as it was.

Specifications cannot be complete and especially regarding things that the program shouldn’t do. It is probably not stated that the software shouldn’t use too much memory or processor for certain operations; it is not stated that the screen shouldn’t flicker, or that all text should be easy to read with all different font settings. Other typical omissions are interactions with other systems; things you expect from all applications under that operating system, internet browser, connected software etc.

You cannot expect a specification to be complete, in most (all? many?) cases, the thing produced by the specification is more important than the document about it. The hardest challenge for the inquisitive tester is to question a lot, but only for those things that are important.

——————–

Who will use the specification? What will they use it for? Will it meet their requirements?

What is it all about? Really?

What areas are left out?

Who is the writer? Does he/she usually miss certain things?

Are there many writers? Does this make the whole less tangible?

Are there many reviewers? Are they using different perspectives?

Is the writer vague, insecure and confusing about certain areas?

Is the specification consistent?

Is the specification consistent with other related specifications?

Is the specification consistent with other different features and combinations of those?

Are all functional and non-functional requirements covered?

Are there dubious thoughts about the wished for functionality?

Are there other sources of information that can be useful?

How is the style of the language affecting the specification?

What quality attributes are the most important, e.g. how is Security weighed against Performance and Usability?

Does it match the system requirements?

Does the specification focus on what is most important?

Does the specification reflect the model of what you think is described?

Are there any new terminology? Will this affect other documentation such as help files?

Is the new terminology consistent with other specifications?

What does the Internet say about the newly chosen terminology? Will there be any misunderstandings?

—————–

If there was no specification, could it be described in a completely different way?

I like to use categorizations to structure my understanding of a subject; and after the simplifications are made and I think I understand it well; the structures can be ripped apart, and you get a bit less confused by the complexity of reality.

There are many forms of requirements, these are some a tester should look out for:

Explicit Requirements
These are the requirements found in the requirement documents. You are probably using them in your testing, making sure that they match the functionality.
This is a quite small part of software testing as I see it.

Implicit Requirements
These are requirements that can be found by combining different requirements that are intertwined.
They could originate from general statements like the program should never crash, or the program should be easy to use, which has implications for many other requirements.
They could also become very large, e.g. support all possible input, or support Python scripting.
They are an effect of vague requirements, and they are a natural part of software development; it would be insane to document everything in advance. Tester can deal with it and understand what is important.

Unspoken Requirements
These are things that many users expect from a program, but they are seldom listed in the requirements document.
Typical examples are behave in same way as other applications on this platform, not leave any garbage files after running, or be appealing to most users.

Incorrect Requirements
The writers of the requirement documents don’t know everything in the world, sometimes they are wrong.
There can be small errors, e.g. inconsistencies between requirements, and huge mistakes, because they didn’t understand the user’s true needs.

Changing Requirements
Sometimes requirements need to be changed, which is something that testers shouldn’t object (too much). The requirements are most likely changed in order to make a better product, and that’s what we all are working for. But when they are changed, or added at a late stage, it can be difficult to challenge them, and test them really good, simply because you are under time-pressure.
We can’t do more than our best, but that’s often enough.

Vague Requirements
I used to dislike vague requirements that were very difficult, almost impossible, to test, but now I think they are good to have.
Not that you should be vague on purpose, but quality attributes like usability, performance etc. can never be detailed and capture the important thing: that customers will be more than satisfied.
It gives you a challenge as a tester; you need to use your feelings and imagination to come up with test ideas, and results that point to a positive or negative indication of result. You can’t hide between some numbers, and must stand for if you think the requirement is met or not.

Hype Requirements
These can be difficult to handle. Often they come in the shape of specifying too much detail, e.g. save settings in an XML file, just because XML is hype (this was 10 years ago, so replace with SOA or the cloud or the hype your company believes in, right now.)
They might be out-of-place, put there in order to be allowed to start the project, but they can also be important, exploiting the hype, or just being a perfect match for this specific application.
As a tester, it’s often not much more to do than accept the hyped requirements, especially if it is accepted (or initiated) by the developers.
But probably you need to learn more about the hype, often there are (at least some) good things inside them.

And regardless of how well all these categories of requirements are implemented and tested, will the application be really, really, super good?

In order to become a successful inquisitive tester, there are a couple of things you can do to improve your skills beyond the more common quest to “question a product”. One important thing is to question the tests themselves.

——————–

Have you ever run tests and wondered if they were really necessary, perhaps knowing that the tests are useless?
Have you ever run tests that were too old and not updated with latest terminology or functionality?
Do your tests contain too much configuration information? Should this be put elsewhere?
Have tests become redundant because those failures no longer happen, ever?
Have the intent of the test been lost and therefore been rendered useless?
Have the tests already been run, on the same build? Twice?
Have you found any bugs or any important information at all with the tests?

Are your tests the most powerful?
Are the tests credible?
Can the tests be faster to execute?
Can you run the most important tests first?
Are the tests too narrow and/or too general?
Do you really understand the test?
Have the original test idea been lost in translation?
Are the tests too much of a projection of the test designer’s thought of view?

Are your tests interesting or boring to execute?
Are the tests in line with your test strategies?
How often do you change your test approaches?
Can the tests instead be better used as input to developers’ unit tests?
Can the essence of the tests be used elsewhere?
Have your tests been reviewed by your colleagues, including technical writers?
Have your scenario tests been reviewed by business people?
Have you captured how the users will use the software in the tests?

Are you satisfied with your tests?
Are your (hidden) stakeholders satisfied with your tests?

——————–

Can you come up with more questions?
Regards,
the test eye (Henrik, Martin & Rikard)