The Quality Status Reporting Fallacy Henrik Emilsson
But wait a minute!
If testers can report the current quality status based on the definition above, it means that test cases corresponds to the requirements; and bugs found are violations where the product characteristics does not satisfy the requirements. If so, then you must have requirements that follow a couple of truths:
- Each requirement should exhibit the statements: Correct, Feasible, Necessary, Prioritized, Unambiguous and Verifiable.
- The set of requirements cover all aspects of people needs.
- The set of requirements capture all people expectations.
- The set of requirements corresponds to the different values that people have.
- The set of requirements contains all the different properties that people value.
- The set of requirements are consistent.
(The word People above include: Users, customers, persons, stakeholders, hidden stakeholders, etc.)
At the same time, we know that it is impossible to test everything; you cannot test exhaustively.
But assume, for the sake of argument, that all requirements were true according to the list above; and the testing was really, really extensive; and the test effort was prioritized so that all testing done was necessary and related to the values that the important stakeholders and customers cared about.
If this would be the case, then how can you compare one test case to another? How can you compare two bugs? Is it possible to compare two bugs even if you have 20 grades of severity?
We, as testers, should be subjective; we should do our best to try to put ourselves in other people’s situation; we should find out who the stakeholders are and what they value; we should try to find all problems that matter.
But we should also be careful when we try to report on these matters. And it is not because we haven’t got any clue about the quality of the product, but we should be careful because many times we report on the things that we do that can be quantified and take these as strong indicators of the quality of the product. E.g., number of bugs found, number of test cases run, bugs found per test case, severe bugs found, severe bugs found per test case per week, etc. You know the drill…
If you are using quantitative measurements, you need to figure out what they really mean and how they connect to what really should (or could) be reported.
If you think that “non-technical” people are pleased by getting a couple of digits (hidden in a graph) presented to them, it is like saying: “Since you aren’t a technical person we have translated the words: Done , Not quite done, Competent, Many, Problems, Requirements, Newly divorced, Few, Fixed, Careless, Test cases, Dyslexic, Needs, Workaholic, Lines of code, Overly complex code, Special configuration, Technical debt, Demands, etc, to some numbers and concealed it all in one graph that shows an aggregate value of the quality”.
I think that it is a bit unfair to the so-called non-technical…
Instead, we should use Jerry Weinberg’s definition “Quality is value to some person” in order to realize that quality is not something easy to quantify. Quality is subjective. Quality is value. Quality relates to some person. Quality is something complex, yet it is intuitive in the eyes of the beholder.
Beautiful post!
I think the defenders of the quality measurements would say that they know that the numbers don’t tell the whole truth, but that it is better than nothing.
And if the numbers are accompanied by an analysis with knowledge about details, it might not be too bad.
What we (people who don’t like metrics-controlled software development) need is a good way to report the collective subjectivity from the involved parties.
How good is gut feeling?
How do you build the trust needed?
And isn’t the decísions really made by feelings already??
Thanks Rikard!
I hear you, but how about beginning to call things by their real name? Instead of reporting Quality Status, the reports should be named “Number of test cases executed” and “Number of bugs found” etc. That is a start. 🙂 When some say that “it is better than nothing” to bring false information, I think they are fools…
I agree with you that the decisions are made by feelings already, so therefore there should be a qualitative investigation done accompanied with the metrics (if any). I think that this should be done by interviewing people in the project and people outside the project. As a tester, I can convey my gut feelings and my subjective judgment during the interview and accompany this with appropriate metrics or other information.
Regarding your sentence “What we need is a good way to report the collective subjectivity from the involved parties.”:
As you mention, there is some sort of hidden need for some “report”. But why should we report anything? Why can’t it be the other way around: a good way to collect the collective subjectivity from the involved parties? The person responsible for the quality is the one that should care for getting the information in a way that she can interpret. I.e., if I was responsible for the quality I would perform an investigation and interviewing people. When I find something interesting I want to be able to drill down into that area, and ask more questions. It would be similar to exploratory testing!
The important thing is that I can make decisions based on the information I dig up.
A good topic.
This is closely related to progress and being able to tell when we are ready to release the product. I think we need to question how we today report status in our projects.
Can we give a good answer to seemingly simple questions like “What is your status?” or “Are we on track?”? In the case that we have defined the full test scope from start of the project, with exact number of test cases that need to be run, exactly how many resources we will have and if there are no delays what so ever in any drop… I can answer it with a value such as 89%. But it will not say anything about the actual quality and it will not be any kind of truth about actual progress or actual status.
Can we ask the sweepers in a curling team: “Exactly how many sweeps do you need to put the thingie into the middle?”
[…] see The Quality Status Reporting Fallacy var a2a_config = a2a_config || {}; a2a_config.linkname="Ignoratio elenchi"; […]