quantitative numbers in a world of qualitative feelings

I am not against measurements in general, they can surely be useful. I use length when building things, weight for baking, time for appointments etc.
I often use numbers for various things in my bug reports.

But metrics are something different; metrics are measurement plus value.
“Should have at least 80% code coverage on unit tests” is a typical example, where “peer reviewed and accepted” would give better results.
“2% better defect detection percentage since last release!” says less than conversations with support department.
“95% Pass on test cases” means nothing at all.

The measurements with value cannot judge what is important;
reality is impossible to aggregate;
metrics are dangerous.

There are many good software testers that advocate metrics. At a couple of occassions I have had the chance to have “the talk” with some I respect.
It boils down to the same thinking: metrics must have a lot of context in order to be useful, so much details that I believe you could throw away the numbers, or only have them as a footnote.
This is not done, because management demands numbers, that’s what they are used to base decisions on.
But for software and testing that might not be well applicable.
If kids want candy, it doesn’t mean you have to give it to them.
You give them better things, out of respect and care.

Testing provides information, so you should find out what is important to different people, and give them what they need, not necessarily what they want.
Metrics will hide importance, and also skew the development efforts.
Try to take them out, with surgical precision.

Then you are left with one final catch:
Qualitative information is hard to aggregate, trust is needed.

“Wouldn’t it be cool if we could come up with a Quality Value for our products?”  said a colleague of mine. ”Yes! That would be super!” me and a couple of colleagues answered.

We had a lot of categories and data in our bug system; and perhaps most important was that we had a good data mining tool that enabled us to take the data and transform it by performing calculations and making aggregations of it.

We started out by analyzing the bug data and starting to come up with reasonable and weighted factors that would enable us to quantify the categories: Severity, Priority, Time to fix, Resolution, Bug Type,  etc. Then we constructed an algorithm that would go through all essential data and the result would be a numeric value, i.e. the Quality Value. We decided that the Quality Value should be in somewhere in the span 0-100 and scoring 100 would be the top Quality Value.
When we discovered anomalies in the result, we tuned the algorithm and the quantifiers so that the result would make more sense; and a lot of discussions were about the quantifiers and their impact on the result. After several iterations we started to get some reasonable numbers.

And by now you might have started wondering how we noticed the anomalies and why we could see that the numbers were reasonable? This happened because we already had a perceived value of the products so we were  biased by subjectivity (huh!).

Anyway, when we were satisfied with the numbers we had a marvelous Quality Value for each and all of our products! And I dare to say that we strongly believed in this Quality Value. Of course there was a constant debate on this subject, and we fought over how much certain categories should have impact on the overall value.
“My product” scored in the top interval so I was very pleased.
But pretty soon, somewhere inside of me, I heard a little voice whisper: “Ignoratio elenchi, ignoratio elenchi, …”

We were of course very naïve in that we believed that this metric would represent the quality of the product. Of course it didn’t!
A couple of observations:

• Bug data only deals with reported bugs
• Bugs are subjective
• Bug reporting is subjective
• Bug handling (management) is subjective
• Bug fixing is subjective
• All other quality criteria not caught in bug reports are not included in bug data
• Quality is value to many people that haven’t reported anything
• Bug data is only data about bugs (+ subjectivity)
• All of the above means that we really cannot compare bugs with each other

On the other hand, one conclusion we came up with that might be true was that the ability to care about the product and bugs were reflected in the Quality Value. And in some way, this meant that a high score indicated that the product was taken care of (see Broken window theory and quality). While this might have been true, we were ever so wrong with the idea on capturing the Product Quality in a single Value…

Read about Ignoratio elenchi

Also see The Quality Status Reporting Fallacy

But wait a minute!

If testers can report the current quality status based on the definition above, it means that test cases corresponds to the requirements; and bugs found are violations where the product characteristics does not satisfy the requirements. If so, then you must have requirements that follow a couple of truths:

• Each requirement should exhibit the statements: Correct, Feasible, Necessary, Prioritized, Unambiguous and Verifiable.
• The set of requirements cover all aspects of people needs.
• The set of requirements capture all people expectations.
• The set of requirements corresponds to the different values that people have.
• The set of requirements contains all the different properties that people value.
• The set of requirements are consistent.

(The word People above include: Users, customers, persons, stakeholders, hidden stakeholders, etc.)
At the same time, we know that it is impossible to test everything; you cannot test exhaustively.

But assume, for the sake of argument, that all requirements were true according to the list above; and the testing was really, really extensive; and the test effort was prioritized so that all testing done was necessary and related to the values that the important stakeholders and customers cared about.
If this would be the case, then how can you compare one test case to another? How can you compare two bugs? Is it possible to compare two bugs even if you have 20 grades of severity?

We, as testers, should be subjective; we should do our best to try to put ourselves in other people’s situation; we should find out who the stakeholders are and what they value; we should try to find all problems that matter.
But we should also be careful when we try to report on these matters. And it is not because we haven’t got any clue about the quality of the product, but we should be careful because many times we report on the things that we do that can be quantified and take these as strong indicators of the quality of the product. E.g., number of bugs found, number of test cases run, bugs found per test case, severe bugs found, severe bugs found per test case per week, etc. You know the drill…

If you are using quantitative measurements, you need to figure out what they really mean and how they connect to what really should (or could) be reported.

If you think that “non-technical” people are pleased by getting a couple of digits (hidden in a graph) presented to them, it is like saying: “Since you aren’t a technical person we have translated the words:  Done , Not quite done, Competent, Many, Problems, Requirements, Newly divorced, Few, Fixed, Careless, Test cases, Dyslexic, Needs, Workaholic, Lines of code, Overly complex code, Special configuration, Technical debt, Demands, etc, to some numbers and concealed it all in one graph that shows an aggregate value of the quality”.

I think that it is a bit unfair to the so-called non-technical…

Instead, we should use Jerry Weinberg’s definition “Quality is value to some person” in order to realize that quality is not something easy to quantify. Quality is subjective. Quality is value. Quality relates to some person. Quality is something complex, yet it is intuitive in the eyes of the beholder.

Recently in Sweden there was a tragic death to a young child that could have been rescued if only the child had come to a hospital in time for a full exam. The one that was blamed for this death was the medical care hotline company that did not understand the severity of the illness and did not send this kid to the hospital. (Read more, in Swedish: http://www.dn.se/sthlm/pojke-dog-efter-rad-att-inte-aka-till-sjukhuset-1.859096)

After this tragic accident, it was discovered that this private medical care hotline company pays out a monthly bonus to those nurses that keep their phone calls short. (Read more, in Swedish: http://www.dn.se/sthlm/skoterskor-far-bonus-for-snabba-rad-1.864534 )
I.e. if they keep the call below 3.48 minutes and during that time complete the medical record, they receive a bonus of 1000 Swedish kronor (approx. € 100). In order to receive the bonus, there are some quality goals as well. E.g., you don’t get the bonus if you unnecessarily send someone to the emergency ward; or if you give a faulty medical advice.
Do I need to tell you that the county council paid the private company by the number of calls they handled.

This is what happens when you use simplified and dangerous metrics as a foundation for incentive pay… And these metrics are easy to abuse because they are based on simplified models of how the real world looks like.
When dealing with people, you are dealing with “complex systems” (read more in An Introduction to General Systems Thinking, by Gerald M. Weinberg ) and you cannot treat every person like they would be the same. I.e., the people calling in (and indeed children that cannot speak for themselves) are treated as a neutral “* 1” or “+ 0” in the metrics equation.
This happens if you include simplified metrics to measure your efficiency when dealing with people; metrics that leaves out the most important and complex parts of the equation: humans and human interaction.
Nurses know how to work with people, they know that people are unique; they know that their job is hard and requires skill and years of experience. They know that some patients require 20 minutes before they are calm or they need such time to explain everything important; they also know that some people just need 25 seconds before they are satisfied.
It is a shame that nurses are measured by how fast they finish a phone call.

It is the same thing that happens again and again in software industry; or rather the peopleware industry. People that work with developing software are measured by metrics that are dangerous and wrong; and in many cases it can have the same tragic outcome as with the young boy that did not reach the hospital in time…

Read more about (dangerous) metrics in the Software Industry:
Software Engineering Metrics: What Do They Measure and How Do We Know?, by Cem Kaner.
Metrics, Schmetrics, by Matthew Heusser.
Meaningful Metrics, by Michael Bolton
Measurements/Metrics/Analysis/Judgment, by Rikard Edgren

At www.context-driven-testing.com you can read “Metrics that are not valid are dangerous.”
I believe this is true, but I would rather prefer “Metrics are dangerous.”

Uninterpreted measurements are not bad by themselves, but when value is added to them, they become metrics, and dangerous because they state specific things without considering a lot of other things, that actually might be much more important.
If measurements are used together with knowledge of details, you might have an analysis that is fruitful.
But at many times, sound judgment not only is enough; it is better.

Measurements/Metrics are probably useful for dead things, like manufacturing objects, but m/m are not good for complex things involving people, e.g. software testing. Metrics uses numbers to reduce the complexity and thereby the “truth” disappears.