I think quality criteria/factors/attributes/characteristics are extremely powerful.
It helps you think in different ways, and makes it easy to get a broader coverage of your test ideas.
See Software Quality Models and Philosophies for McCall, Boehm, FURPS, Dromey, ISO 9126 models, or CRUSSPIC STMPL for a version without focus on measurability.
The granularity of this (and all other) categorization can be discussed, so here are suggested sub-categories for Capability, with some thoughts for inspiration:

Capability – The set of bigger and smaller things you can accomplish with the software. This is usually covered by requirements or similar, and with the addition of some help from developers telling about small or hidden features, you can cover this with thorough and hard work. I suspect some test efforts stop here.
The most lightweight testing approach is to ignore this totally, it has been considered by others, and you will cover parts in more interesting ways when performing other testing.

Completeness – Are the functionality really enough? Or are there small things in between, and on the edges that are needed to create a killer app? As a tester you won’t decide the scope for a release, but you can identify small things that can be implemented almost for free, and since you have knowledge of the system you can identify bigger things that someone else can put up on the list.
Lightweight method: think about what is missing while performing system testing.

Correctness – This could be seen as an obvious part of capability, but if you think about precision for real/double, or other corner cases, there are a lot to investigate, and probably a lot to ignore as well…
I don’t know about any lightweight technique for this, except asking detailed questions about what’s important.

Efficiency – Does the product do what it is supposed to do in an effective manner, without doing what it isn’t supposed to do?
Lightweight: keep your eyes open and look at more places than the apparent ones.

Interoperability – In a requirements document you will find a lot of things the software should be able to do, but you will not get a list of all important combinations and interactions that surely exists. Pairwise testing is a theoretical solution to this dilemma (and I guess it is effective for some situations), but with knowledge about the product you will see that some combinations are more error-prone than others.
A lightweight testing solution consists of two test ideas: 1) turn on everything 2) turn off everything

Concurrency – Can the software or functions therein operate simultaneously? How many concurrent actions? What if they are dependent on each other?
Lightweight testing: start more operations now and then while system testing.

Extendability – all features wanted by customers can’t be implemented, so it can be nifty with an API that allows extensions of various types.
Lightweight testing: get hold of an API implementation, use it and change it.

What is wrong, what is missing?

“Wouldn’t it be cool if we could come up with a Quality Value for our products?”  said a colleague of mine. ”Yes! That would be super!” me and a couple of colleagues answered.

We had a lot of categories and data in our bug system; and perhaps most important was that we had a good data mining tool that enabled us to take the data and transform it by performing calculations and making aggregations of it.

We started out by analyzing the bug data and starting to come up with reasonable and weighted factors that would enable us to quantify the categories: Severity, Priority, Time to fix, Resolution, Bug Type,  etc. Then we constructed an algorithm that would go through all essential data and the result would be a numeric value, i.e. the Quality Value. We decided that the Quality Value should be in somewhere in the span 0-100 and scoring 100 would be the top Quality Value.
When we discovered anomalies in the result, we tuned the algorithm and the quantifiers so that the result would make more sense; and a lot of discussions were about the quantifiers and their impact on the result. After several iterations we started to get some reasonable numbers.

And by now you might have started wondering how we noticed the anomalies and why we could see that the numbers were reasonable? This happened because we already had a perceived value of the products so we were  biased by subjectivity (huh!).

Anyway, when we were satisfied with the numbers we had a marvelous Quality Value for each and all of our products! And I dare to say that we strongly believed in this Quality Value. Of course there was a constant debate on this subject, and we fought over how much certain categories should have impact on the overall value.
“My product” scored in the top interval so I was very pleased.
But pretty soon, somewhere inside of me, I heard a little voice whisper: “Ignoratio elenchi, ignoratio elenchi, …”

We were of course very naïve in that we believed that this metric would represent the quality of the product. Of course it didn’t!
A couple of observations:

• Bug data only deals with reported bugs
• Bugs are subjective
• Bug reporting is subjective
• Bug handling (management) is subjective
• Bug fixing is subjective
• All other quality criteria not caught in bug reports are not included in bug data
• Quality is value to many people that haven’t reported anything
• Bug data is only data about bugs (+ subjectivity)
• All of the above means that we really cannot compare bugs with each other

On the other hand, one conclusion we came up with that might be true was that the ability to care about the product and bugs were reflected in the Quality Value. And in some way, this meant that a high score indicated that the product was taken care of (see Broken window theory and quality). While this might have been true, we were ever so wrong with the idea on capturing the Product Quality in a single Value…

Read about Ignoratio elenchi

Also see The Quality Status Reporting Fallacy

I grew up in a small “town” in Värmland. Outside the village, most people live in isolated houses/farms on the countryside or in the woods.
If you’d ask one of these persons what quality is, they would answer:

dä ä väl att fôlk töcker att dä ä bra
(guess it’s that people like it)

This is actually quite a nice summary, well in line with Jerry Weinbergs “quality is value to some person”; but it doesn’t help us a lot as software testers (except clearly pointing at quality being something subjective.)
I read Marlena Compton’s blog post on quality and art (http://marlenacompton.com/?p=605) and realized that quality, like (good) art, isn’t so hard to distinguish when you see it, at least not if it’s quality for yourself. So here’s a try at a slightly more fruitful definition for testers:

quality is value to me, people I know, or people I don’t know

If I am a customer, or am using the software with a customer’s understanding and needs, it is not difficult to understand what is important and needs to be fixed.
If I know a lot about the customers and the area where the product is operating, it is also not so difficult to evaluate if the software is “good”.
If I don’t know the users, I can only take guesses at what quality could be for the product.
So as testers we should try to get more “Me” people, maybe by using the product internally (eat our own dog food, drink our own champagne/pommac), and we should try to move “Don’t Know” to “Know” by learning, using Personas, learning, visiting customers, learning etc.

But when reading TASSQ magazine from 2006 (http://www.tassq.org/quarterly/docs/TASSQuarterly0604B.pdf) there were some definitions that said that we don’t have to be so subjective in the quality definition, e.g. if it is possible to say that a car with certain attributes is of high quality, why can’t we do it for software.
This leads me to another try:

quality is more than the subjective sum of relevant quality attributes like
capability, trustworthiness, usability, security, performance, compatibility, entertaining…

You can use many, many different quality attributes, and the importance will differ for products and types of users, but by thinking about the right ones, you will get a hint on what would be better or worse quality.
One caution: quality is “more” than the sum of the attributes, there is something unknown and magical that some software has, and some don’t.

All in all; I don’t know what software quality is, but I know that it is very important and something we should continuously think about.
We won’t come up with something universally true, but it can be useful.

Now and then I read that testers don’t know enough about Usability, that there is a need for a Performance Testing expert, that a Security consultant should be called in, or that a master of the used technology would make Installation and Compatibility testing possible.
This might be true in the general case, but there are many testers that are working with the same product suite for years, and over time you become an expert of most aspects of quality for your specific product.

Let’s look at some examples:

One sub-category of Usability is Operability; that an experienced user can perform functions in a fast and efficient way. As a system tester, you have done most operations many times, and you know what you can expect from features in the sense of operability. You can point at places where for instance the ability to delete multiple items with a few clicks would make a difference.
Learnability is another sub-category of Usability. You only learn the basics once, but maybe you have heard customer stories of confusing things, or you could let a new member of the test team think in this direction.
Regarding Accessibility it doesn’t cost too much to use High DPI, speakers turned on, Code Blind or Magnifier sometimes.

Large-scale Performance might require an expensive tool, but I bet you’re doing some simulations without them; maybe just by telling the whole development team to hit the same server at the same time.
There is also a small-scale, low-level Performance aspect that shouldn’t be understimated. If a dialog takes more than a second to display, it might be something that put the user’s confidence in doubt.
If you have tested your product for quite some time, you will immediately notice when something takes just a bit longer than it could take. There of course might be valid reasons for this, but talking to the developer about it might be beneficial to both the producers and consumers of your software.

Security testing seems more difficult, but as a product tester you know at which moments authentication takes place, you know if there are passwords stored, and that they should be encryted; you might not know how to exploit a crash, but you are an expert at provoking the crashes.

For Hardware/OS/Application Compatibility testing you will become more of an expert the longer you work with them, at least if you have the curiousity to learn more things when you get the chance.
You might not know a lot about how the iPhone works, but you know all the details that can be used for interacting with your web site.

Sometimes I also read the extreme that testers shouldn’t bother with Usability/Security/Performance testing, which to me seems like an incredible waste of knowledge and resources.
When testing functionality manually, you can look at quality attributes at the same time, and get a lot of coverage for free.

I’m not saying true specialists aren’t needed, but I’m saying that there are a lot of expertise in your building that at least can be used complementary.
If you are in a situation where you know a lot of these things, but aren’t allowed/encouraged to test these things, I think you should try to convince your managers of a better and more fun way to test your product.

But wait a minute!

If testers can report the current quality status based on the definition above, it means that test cases corresponds to the requirements; and bugs found are violations where the product characteristics does not satisfy the requirements. If so, then you must have requirements that follow a couple of truths:

• Each requirement should exhibit the statements: Correct, Feasible, Necessary, Prioritized, Unambiguous and Verifiable.
• The set of requirements cover all aspects of people needs.
• The set of requirements capture all people expectations.
• The set of requirements corresponds to the different values that people have.
• The set of requirements contains all the different properties that people value.
• The set of requirements are consistent.

(The word People above include: Users, customers, persons, stakeholders, hidden stakeholders, etc.)
At the same time, we know that it is impossible to test everything; you cannot test exhaustively.

But assume, for the sake of argument, that all requirements were true according to the list above; and the testing was really, really extensive; and the test effort was prioritized so that all testing done was necessary and related to the values that the important stakeholders and customers cared about.
If this would be the case, then how can you compare one test case to another? How can you compare two bugs? Is it possible to compare two bugs even if you have 20 grades of severity?

We, as testers, should be subjective; we should do our best to try to put ourselves in other people’s situation; we should find out who the stakeholders are and what they value; we should try to find all problems that matter.
But we should also be careful when we try to report on these matters. And it is not because we haven’t got any clue about the quality of the product, but we should be careful because many times we report on the things that we do that can be quantified and take these as strong indicators of the quality of the product. E.g., number of bugs found, number of test cases run, bugs found per test case, severe bugs found, severe bugs found per test case per week, etc. You know the drill…

If you are using quantitative measurements, you need to figure out what they really mean and how they connect to what really should (or could) be reported.

If you think that “non-technical” people are pleased by getting a couple of digits (hidden in a graph) presented to them, it is like saying: “Since you aren’t a technical person we have translated the words:  Done , Not quite done, Competent, Many, Problems, Requirements, Newly divorced, Few, Fixed, Careless, Test cases, Dyslexic, Needs, Workaholic, Lines of code, Overly complex code, Special configuration, Technical debt, Demands, etc, to some numbers and concealed it all in one graph that shows an aggregate value of the quality”.

I think that it is a bit unfair to the so-called non-technical…

Instead, we should use Jerry Weinberg’s definition “Quality is value to some person” in order to realize that quality is not something easy to quantify. Quality is subjective. Quality is value. Quality relates to some person. Quality is something complex, yet it is intuitive in the eyes of the beholder.

I use Jerry Weinberg’s definition of quality: “Quality is value to some person”; and I use Cem Kaner’s extension to the definition so that it becomes “Quality is value to some person (that matters)”…

I.e. quality is inherently subjective. And there are a lot of persons that are affected by software that we produce… With this in mind it becomes hard for a tester to stay focused when there are so many persons with opinions that could matter; but if we can find out “who matters” we decrease the number of possible values to care about. Still, this will leave us with several important values that need to be taken into account when testing the product.

So how can we testers deal with that?

You could do a role play when testing and put on someone’s hat during the test session; or you could let real users test the product and let them have a say about what they find.
But for a skilled tester it is more about being multidimensionally subjective and think as several persons at the same time.

This means that a lot of values, beliefs and preferences are taken into account which might matter. Not as an average, but as several independent quality dimensions that has (more or less) importance. The hard thing is to know when a value is threatened and for which (type of) person that is affected; and if this matters at all.
I.e., it is a matter of questioning “is there a problem here?” constantly and try to pair a potentially threatened value with its corresponding person. And if this problem threatens a value for some person that matters, we have found a bug. This corresponds to the definition of bug from Cem Kaner “A bug is something that threatens the value of a product”.

Much of this happens automatically for many of you skilled testers out there; when I thought of it recently I realized that this is something I do more and more and hopefully I am improving this skill each day. This is a great skill to have when testing software!

Anyone having any thoughts on this?
Have you experienced this yourself?
If not, does it sound like an interesting thing to examine? Would this be helpful to you?

Cheers,
Henrik

Update 2009-09-14: According to comment from Michael Bolton, see below, the quotes that I said belonged to Cem Kaner are both quotes from James Bach. I apologize for referencing wrong person.

When you come up with a new test idea, you are using your knowledge and experience, but there is also some sort of stimuli that triggers the idea. Something you see, hear, understand or think about.

You seldom think in totally new ways, you rather combine things in a new way.

These are my favorite test idea triggers:

* think about each feature (and connect with testing or technical knowledge)

* create or understand a model of the software

* think about interaction between items in the system

* thinking about quality attributes (CRUSSPIC STMPL + Accessibility)

* read bug report titles for similar functionality

* read test idea lists

* play around with the software, or a prototype, or a previous version

* change perspective, see your software as a little box among many others

What triggers you?

Do you get help from Session Tester Primers?
(http://sessiontester.openqa.org/download.html)

Thare are many ways to look at the problem of testing software, and it is rarely wise to use only one; that’s why there are so many mnemonics, e.g. SFDPOT, CRUSSPIC STMPL, HICCUPPS, FCC CUTS VIDS (http://www.satisfice.com/tools/satisfice-tsm-4p.pdf, http://www.developsense.com/articles/2005-01-TestingWithoutAMap.pdf, http://www.testingreflections.com/node/view/2823)
Here are some questions for your confusion:

1) What?
a) The functionality, what the software does from A to Z; what elements the software works with; the source code, from beginning to end; the requirements, the use cases, the test cases; the product as a whole, including documentation and support. None of these give the full picture.
b) The actual tests, their artifacts, the communication.

2) How?
a) The attributes of the functionality; Usability, Performance, Security, Reliability, Installability, Compatibility, Scalability, Accessibility.
b) The process (both the documented and the real) used when testing; your techniques and approaches, your tools.

3) Who?
a) The (potential) users, which you probably can’t bring down to a couple of personas, but it is good to know what the users are like; what language they speak; what most of them knows.
b) The testers, your (different) skills. Will the developers also do testing? Will customers test the software, in order to accept the product or find bugs?

4) Why?
a) What do you think the users are trying to accomplish with the software? Can the software be used in many more number of ways as there are users? Are there any unintentional effects that are desirable? Why should the user not switch to your competititor?
b) What are the main drivers for the testing effort: Find all relevant bugs; provide enough information for the project to succeed; minimize customer complaints; learn and explore?

5) Where?
a) The environments where the software will operate and interact. Specific hardware and software?
b) Which environments can you afford to use; how far can you get with Basic Configuration Matrix (http://thetesteye.com/blog/2008/05/bcm-basic-configuration-matrix/)

6) When?
a) Will the software only be used at one given time, at one place? Will upcoming releases replace the old version, or is it necessary to be forward compatible with currently non-existant software?
b) When will you test the software, and in which shape will it be?