More Definitions of Quality Rikard Edgren

I grew up in a small “town” in Värmland. Outside the village, most people live in isolated houses/farms on the countryside or in the woods.
If you’d ask one of these persons what quality is, they would answer:

dä ä väl att fôlk töcker att dä ä bra
(guess it’s that people like it)

This is actually quite a nice summary, well in line with Jerry Weinbergs “quality is value to some person”; but it doesn’t help us a lot as software testers (except clearly pointing at quality being something subjective.)
I read Marlena Compton’s blog post on quality and art (http://marlenacompton.com/?p=605) and realized that quality, like (good) art, isn’t so hard to distinguish when you see it, at least not if it’s quality for yourself. So here’s a try at a slightly more fruitful definition for testers:

quality is value to me, people I know, or people I don’t know

If I am a customer, or am using the software with a customer’s understanding and needs, it is not difficult to understand what is important and needs to be fixed.
If I know a lot about the customers and the area where the product is operating, it is also not so difficult to evaluate if the software is “good”.
If I don’t know the users, I can only take guesses at what quality could be for the product.
So as testers we should try to get more “Me” people, maybe by using the product internally (eat our own dog food, drink our own champagne/pommac), and we should try to move “Don’t Know” to “Know” by learning, using Personas, learning, visiting customers, learning etc.

But when reading TASSQ magazine from 2006 (http://www.tassq.org/quarterly/docs/TASSQuarterly0604B.pdf) there were some definitions that said that we don’t have to be so subjective in the quality definition, e.g. if it is possible to say that a car with certain attributes is of high quality, why can’t we do it for software.
This leads me to another try:

quality is more than the subjective sum of relevant quality attributes like
capability, trustworthiness, usability, security, performance, compatibility, entertaining…

You can use many, many different quality attributes, and the importance will differ for products and types of users, but by thinking about the right ones, you will get a hint on what would be better or worse quality.
One caution: quality is “more” than the sum of the attributes, there is something unknown and magical that some software has, and some don’t.

All in all; I don’t know what software quality is, but I know that it is very important and something we should continuously think about.
We won’t come up with something universally true, but it can be useful.

12 Comments
Saam February 11th, 2010

I dont know what software quality is either! :) But still I agree it is important.

Some thoughts I have regarding this subject: If quality is something subjective then we must fulfill the expectations and opinions of all (important) users when building our software in order to achieve “true quality”, and that is not normally feasible (consider also that many expectations are likely to be contradictory). In this sense I dont think any company is aiming for “true quality”. Instead we group, summarize, and make assumptions in order to get realistic targets that we can measure and still believe will provide “good enough quality”. I think by doing this we are actually trying to make quality into something objective rahter than subjective. Perhaps there is room for both a subjective-orinented definition and an objective-oriented definition of quality? Am I allowed to accept two (even contradictory) definitions of the same word?

Rikard Edgren February 11th, 2010

Of course we can use multiple definitions, but it should be explicit for good communication.

I think there is merit to your idea that we try to make a assumptions in order to get realistic regarding quality in the project.
For a company or a project one could define quality objectives (e.g. never crash more than once a day, follow MacOS look & feel, faster than competitor X etc.),
and state that they are the synthesis of the quality you think is most important.
One could also mention that “true quality” is more than this, so use your subjectivity when designing, implementing and testing.

You can aim for “true quality”, but know you won’t reach it for every user.
The objective definition is a pragmatic approximation of quality.

Henrik Emilsson February 11th, 2010

Why don’t stick with the Kaner and Bach extension to Weinbergs definition?
“Quality is value to someone (that matters)”. This helps you as a software tester more than the original definition. And the key here is of course “that matters” (sometimes it could also be spelled out as “important enough”) and the importance of finding out who these someones are.

There is also a difference between using Quality the noun and Quality the adjective. E.g., “Quality Software” is software of superior quality and has the (quality) properties that matters for its users. “Software Quality” are those value properties that matters for software and its intended users.

If you talk about quality as an adjective, then you already know what properties that are important for a kind of product. I.e., the product could be measured against other products of similar kind and you can know by just looking at it that it looks like it has good quality. And of course software can also be measured like this, but since cars are a solution to similar problems and software are solutions to a very broad range of problems I guess that the criterion that we use for software is harder to define. So I think you could use your last definition that tries to define the mysterious qualitative properties that surround software; i.e. the criterion of quality software.

[...] others” is when I see it, if I know a lot about the intended usage and users.” Rikard also wrote a post where he clarifies his position a [...]

Rikard Edgren February 13th, 2010

Henrik; I have never liked the “(that matters)” addition. Don’t really know why, it’s not anything wrong with it, maybe I (incorrectly?) think that it implies that some people are more worth than others (which makes sense from the product’s perspective…)
I use quality as a noun, and I think pragmatic and useful definitions are best created for specific products, to be used for better design, implementation and testing.

Martin Jansson February 13th, 2010

I think it is good to redefinition your beliefs, it makes you change your thinking and assumptions.

Still, I am fond of using “Quality is value to someone that matters” because with just that frase I can ask a load of questions about my test objective, scope and so on. I use that definition as a tool in understanding what I am supposed to do as a tester.

If I were to change the definition I might identify a new set of questions.

When you work with a complex system, where you as a tester can only phantom a small part of what kinds of users there are, I do not think a single tester can say “I know what the value is”. You might know something that someone values, but you will not be able to know all values. If you have great domain knowledge and great experience as a tester you will naturally have it easier.

Saam February 14th, 2010

Regarding “Quality is value to someone that matters”. Henrik, you wrote “This helps you as a software tester “. Martin, you write ” I use that definition as a tool in understanding what I am supposed to do as a tester”. I think thats where this definition works best, on an individual or small scale level (high degree of influence on the output) with limited or known stakeholders. However if we want to build software with high quality in a complex environment where you deliver your output to a large amount of different users and a large amount of people are involved in compiling that output, then we today need an approach that “is a pragmatic approximation of quality” (nicely put Rikard!) that we can align around, use for clear communication, track progress on, and measure.

Saam February 14th, 2010

A comment on my own comment: maybee I compare apples with pears. But my intention was to highlight the need of an objective view of quality (ofcourse specific to our current context), as I believe that is more productive than a subjective view when it comes to “complex environments”.

Henrik Emilsson February 14th, 2010

I do not think that we ever will know what the sum of quality is, that will be really hard. The more complex the product is and/or the much diversified a user group is, the harder it gets to know what the “sum of quality” is.
But if you know that “quality is value to someone that matters” you can understand that one user might value just one thing, and another user values a completely different thing. This is a humble, vague and pragmatic fact. But finding out this for all users is as much impossible as to test exhaustively. So you have to choose, as you do when you test. And focusing on those users that matter is the same thing as to test those things that matter.

To say that “quality is more than the subjective sum of relevant quality attributes like capability, trustworthiness, usability, security, performance, compatibility, entertaining…” is like saying that there are more things to test than meets the eye… Which is true! And knowing this is good, but it is rather useless if you are going to use it as a guidance in your daily work.

Rikard Edgren February 14th, 2010

Maybe we need two categories of definitions, one that tries to be “true”, and one that tries to be “useful”.
For the “true” definition, I think the key is in Kierkegaard’s “the truth is the subjectivity”, which is nicely captured in Weinberg’s “quality is value to some person”.
For the “useful” definitions, it will differ for different people, depending on what they do with it. “that matters” seem to trigger good questions for some people; quality attributes work fine for me as a reminder of what to think about; and I also think they can be valuable in (complex) environments, e.g. if used to define quality objectives.
Maybe the “that matters” definítion is better for products with clear target users and/or strong stakeholders, whereas quality attributes are more appropriate for an “off-the-shelf” product where it is more difficult to anticipate the different types of users the product will interact with.

Martin Jansson February 14th, 2010

When thinking of definitions also consider the cost of quality as in
http://www.kaner.com/pdfs/Quality_Cost_Analysis.pdf, to put it into another context.

Will this change the way you define quality? If not, can you define quality without putting it in the context where we most talk about it?

Henrik Emilsson February 15th, 2010

Perhaps my latest reply needs a clarification:

Using quality attributes as inspiration for test ideas is a very good thing to do since they are a set of categorized indicators on quality candidates that could represent subjective values from users and/or project stakeholders.
But using these as a definition of quality is misleading.

In order for these to be truly useful you need to know a lot about your users, stakeholders and the products. Then you know What matters; Who matters; and How it matters; and can use it in your daily work as a definition of your product quality.