If you haven’t done it before, have a look at the Software Quality Characteristics that we published last year: TheTestEye – Software Quality Characteristics

You can probably imagine ways of testing for all of these quality characteristics yourself, and you might even come up with good oracles that can assist you in the interpretation of the test results. However, the Charisma characteristics might be the most subjective ones; and it might also be a case of where it is really important to find out, for each quality characteristic respectively, which stakeholder whose values matters the most.

So how do you test for a product’s Charisma?

Charisma. Does the product have “it”?

• Satisfaction: how does it feel after using the product?
• Professionalism: does the product have the appropriate flair of professionalism and feel fit for purpose?
• Attractiveness: are all types of aspects of the product “good-looking”?
• Curiosity: will users get interested and try out what they can do with the product?
• Entrancement: do users get hooked, have fun, in a flow, and fully engaged when using the product?
• Hype: does the product use too much or too little of the latest and greatest technologies/ideas?
• Expectancy: the product exceeds expectations and meets the needs you didn’t know you had.
• Attitude: do the product and its information have the right attitude and speak to you with the right language and style?
• Directness: are (first) impressions impressive?
• Story: are there compelling stories about the product’s inception, construction or usage?

All of these are somewhat intuitive and I guess that you, consciously or unconsciously, test for these everyday in your project. You can also focus on testing these in the same fashion as when testing for other quality characteristics.
However, if (some of) these software quality characteristics really matter in your project and are core values for your product, you better question your own capability of being able to test this.
By question yourself; you might realize that your oracles aren’t powerful enough in order to really being able to answer the question “Is there a problem here”.
You might be too biased; or you might not be able to really understand what the stakeholders need or value. Or you might simply be too unimportant in order to even “have opinions”…

One way of addressing this is by letting a proxy user act as observer and interpreter of the test result. This can be a good approach when testing for software quality characteristics where you suspect that your own oracles won’t be enough in order to understand what is important. Well this isn’t news for many of you, since user testing has been used successfully for years. But my point is that you can spend some extra thoughts on which you select as your proxy oracle – with a mission to test for those selected Charisma values.

You can do this kind of testing by letting a proxy sit next to you while you test; or by letting users test the program themselves and you interviewing them afterwards. Another way can be to test or demo in front of an audience of proxies that observe and interpret the result as you go.

The tests could be designed as open questions that you would have asked the program and watched it respond, but instead you ask the questions to the proxies and letting them answer in the way they interpret the result.

We all test as children; we are curious and want to find things out, before we are one year old we want to break things, and after three we ask “What If” questions.
My first ambitious testing journey came many years later.

Still a naive teenager, I started studying Philosophy at the university. It was an easy choice; philosophy was the subject where you could find the most essential truths.
It took me one and a half year of testing philosophical theories to find out I was wrong.

The classes went through the history of philosophy, and each theory was heavily criticized by me (and a classmate), not because I was a stubborn skeptic, but because I was an objectivist looking for real truth.
I met some very interesting philosophers (Heraclites, Spinoza and Wittgenstein being favorites), but the theories did not stand the test; there were flawed arguments and logic holes alongside with incorrect or unjustified assumptions.
I was disappointed, and finally realized everything is grounded in assumptions you can choose to agree with or not. (I’m OK with this now, I even see it as the essence of the charm of life.)
It was not wasted time, this massive falsification was a hard school in logical and artificial thinking, which are key skills for software testers.

A couple of years later I found the best philosophical theory at Kierkegaard (don’t stumble on his Christian dilemmas) who said that “the truth is the subjectivity”.
He did not state that the objective science was wrong, just that they didn’t grasp what is important.
To be human is to be subjective.

So whenever I rant about traditional testing theory, or coverage, or metrics, or expected results, I don’t say that they are wrong; they just don’t capture what is important.
If software is made for humans; testers should use their subjectivity.

“Wouldn’t it be cool if we could come up with a Quality Value for our products?”  said a colleague of mine. ”Yes! That would be super!” me and a couple of colleagues answered.

We had a lot of categories and data in our bug system; and perhaps most important was that we had a good data mining tool that enabled us to take the data and transform it by performing calculations and making aggregations of it.

We started out by analyzing the bug data and starting to come up with reasonable and weighted factors that would enable us to quantify the categories: Severity, Priority, Time to fix, Resolution, Bug Type,  etc. Then we constructed an algorithm that would go through all essential data and the result would be a numeric value, i.e. the Quality Value. We decided that the Quality Value should be in somewhere in the span 0-100 and scoring 100 would be the top Quality Value.
When we discovered anomalies in the result, we tuned the algorithm and the quantifiers so that the result would make more sense; and a lot of discussions were about the quantifiers and their impact on the result. After several iterations we started to get some reasonable numbers.

And by now you might have started wondering how we noticed the anomalies and why we could see that the numbers were reasonable? This happened because we already had a perceived value of the products so we were  biased by subjectivity (huh!).

Anyway, when we were satisfied with the numbers we had a marvelous Quality Value for each and all of our products! And I dare to say that we strongly believed in this Quality Value. Of course there was a constant debate on this subject, and we fought over how much certain categories should have impact on the overall value.
“My product” scored in the top interval so I was very pleased.
But pretty soon, somewhere inside of me, I heard a little voice whisper: “Ignoratio elenchi, ignoratio elenchi, …”

We were of course very naïve in that we believed that this metric would represent the quality of the product. Of course it didn’t!
A couple of observations:

• Bug data only deals with reported bugs
• Bugs are subjective
• Bug reporting is subjective
• Bug handling (management) is subjective
• Bug fixing is subjective
• All other quality criteria not caught in bug reports are not included in bug data
• Quality is value to many people that haven’t reported anything
• Bug data is only data about bugs (+ subjectivity)
• All of the above means that we really cannot compare bugs with each other

On the other hand, one conclusion we came up with that might be true was that the ability to care about the product and bugs were reflected in the Quality Value. And in some way, this meant that a high score indicated that the product was taken care of (see Broken window theory and quality). While this might have been true, we were ever so wrong with the idea on capturing the Product Quality in a single Value…

Read about Ignoratio elenchi

Also see The Quality Status Reporting Fallacy

But wait a minute!

If testers can report the current quality status based on the definition above, it means that test cases corresponds to the requirements; and bugs found are violations where the product characteristics does not satisfy the requirements. If so, then you must have requirements that follow a couple of truths:

• Each requirement should exhibit the statements: Correct, Feasible, Necessary, Prioritized, Unambiguous and Verifiable.
• The set of requirements cover all aspects of people needs.
• The set of requirements capture all people expectations.
• The set of requirements corresponds to the different values that people have.
• The set of requirements contains all the different properties that people value.
• The set of requirements are consistent.

(The word People above include: Users, customers, persons, stakeholders, hidden stakeholders, etc.)
At the same time, we know that it is impossible to test everything; you cannot test exhaustively.

But assume, for the sake of argument, that all requirements were true according to the list above; and the testing was really, really extensive; and the test effort was prioritized so that all testing done was necessary and related to the values that the important stakeholders and customers cared about.
If this would be the case, then how can you compare one test case to another? How can you compare two bugs? Is it possible to compare two bugs even if you have 20 grades of severity?

We, as testers, should be subjective; we should do our best to try to put ourselves in other people’s situation; we should find out who the stakeholders are and what they value; we should try to find all problems that matter.
But we should also be careful when we try to report on these matters. And it is not because we haven’t got any clue about the quality of the product, but we should be careful because many times we report on the things that we do that can be quantified and take these as strong indicators of the quality of the product. E.g., number of bugs found, number of test cases run, bugs found per test case, severe bugs found, severe bugs found per test case per week, etc. You know the drill…

If you are using quantitative measurements, you need to figure out what they really mean and how they connect to what really should (or could) be reported.

If you think that “non-technical” people are pleased by getting a couple of digits (hidden in a graph) presented to them, it is like saying: “Since you aren’t a technical person we have translated the words:  Done , Not quite done, Competent, Many, Problems, Requirements, Newly divorced, Few, Fixed, Careless, Test cases, Dyslexic, Needs, Workaholic, Lines of code, Overly complex code, Special configuration, Technical debt, Demands, etc, to some numbers and concealed it all in one graph that shows an aggregate value of the quality”.

I think that it is a bit unfair to the so-called non-technical…

Instead, we should use Jerry Weinberg’s definition “Quality is value to some person” in order to realize that quality is not something easy to quantify. Quality is subjective. Quality is value. Quality relates to some person. Quality is something complex, yet it is intuitive in the eyes of the beholder.

At EuroStar 2008 I presented Testing is an Island – A Software Testing Dystopia.
Fritz shot the pictures, Henrik wrote the music, and I uploaded it on YouTube:

The accompanying paper can be found at http://www.thetesteye.com/papers/redgren_testingisanisland.doc

I use Jerry Weinberg’s definition of quality: “Quality is value to some person”; and I use Cem Kaner’s extension to the definition so that it becomes “Quality is value to some person (that matters)”…

I.e. quality is inherently subjective. And there are a lot of persons that are affected by software that we produce… With this in mind it becomes hard for a tester to stay focused when there are so many persons with opinions that could matter; but if we can find out “who matters” we decrease the number of possible values to care about. Still, this will leave us with several important values that need to be taken into account when testing the product.

So how can we testers deal with that?

You could do a role play when testing and put on someone’s hat during the test session; or you could let real users test the product and let them have a say about what they find.
But for a skilled tester it is more about being multidimensionally subjective and think as several persons at the same time.

This means that a lot of values, beliefs and preferences are taken into account which might matter. Not as an average, but as several independent quality dimensions that has (more or less) importance. The hard thing is to know when a value is threatened and for which (type of) person that is affected; and if this matters at all.
I.e., it is a matter of questioning “is there a problem here?” constantly and try to pair a potentially threatened value with its corresponding person. And if this problem threatens a value for some person that matters, we have found a bug. This corresponds to the definition of bug from Cem Kaner “A bug is something that threatens the value of a product”.

Much of this happens automatically for many of you skilled testers out there; when I thought of it recently I realized that this is something I do more and more and hopefully I am improving this skill each day. This is a great skill to have when testing software!

Anyone having any thoughts on this?
Have you experienced this yourself?
If not, does it sound like an interesting thing to examine? Would this be helpful to you?

Cheers,
Henrik

Update 2009-09-14: According to comment from Michael Bolton, see below, the quotes that I said belonged to Cem Kaner are both quotes from James Bach. I apologize for referencing wrong person.

At EuroSTAR 2008 in Haag I will present “Testing is an Island – A Software Testing Dystopia”. The paper can be downloaded at http://www.thetesteye.com/papers/redgren_testingisanisland.doc

The inspiration was the theme of the conference: “the future of software testing”; and I couldn’t stop seeing a very boring profession, where numbers and so-called objectivity is more important than people and feelings.
There are trends and silver bullets that miss our secret: we perform simultaneous verification and validation.