Multidimensional Subjectivity in Software Testing Henrik Emilsson
I use Jerry Weinberg’s definition of quality: “Quality is value to some person”; and I use Cem Kaner’s extension to the definition so that it becomes “Quality is value to some person (that matters)”…
I.e. quality is inherently subjective. And there are a lot of persons that are affected by software that we produce… With this in mind it becomes hard for a tester to stay focused when there are so many persons with opinions that could matter; but if we can find out “who matters” we decrease the number of possible values to care about. Still, this will leave us with several important values that need to be taken into account when testing the product.
So how can we testers deal with that?
You could do a role play when testing and put on someone’s hat during the test session; or you could let real users test the product and let them have a say about what they find.
But for a skilled tester it is more about being multidimensionally subjective and think as several persons at the same time.
This means that a lot of values, beliefs and preferences are taken into account which might matter. Not as an average, but as several independent quality dimensions that has (more or less) importance. The hard thing is to know when a value is threatened and for which (type of) person that is affected; and if this matters at all.
I.e., it is a matter of questioning “is there a problem here?” constantly and try to pair a potentially threatened value with its corresponding person. And if this problem threatens a value for some person that matters, we have found a bug. This corresponds to the definition of bug from Cem Kaner “A bug is something that threatens the value of a product”.
Much of this happens automatically for many of you skilled testers out there; when I thought of it recently I realized that this is something I do more and more and hopefully I am improving this skill each day. This is a great skill to have when testing software!
Anyone having any thoughts on this?
Have you experienced this yourself?
If not, does it sound like an interesting thing to examine? Would this be helpful to you?
Cheers,
Henrik
Update 2009-09-14: According to comment from Michael Bolton, see below, the quotes that I said belonged to Cem Kaner are both quotes from James Bach. I apologize for referencing wrong person.
Hi,
Good testing skills in addition to strong domain knowledge is a strong combo when finding problems that matters (with a reasonble effort = quickly). Hence the importance of having members with different skills/knowledge in your testing team I think.
/Stefan
@Stefan,
One interesting thing with “when finding problems that matters” is that in many cases the bug did not matter when it was found, but a bit later it was a lot more important. So what matters is ever changing, not to be thought of as a permanent thing.
Absolutely.
All this is at the heart of the Rapid Software Testing methodology that James Bach and I teach. It’s certainly at the heart of the Heuristic Test Strategy Model (HTSM; http://www.satisfice.com/tools/htsm.pdf ). It’s the motivation for Operations, one of the product elements in the HTSM. It’s the motivation for User testing and Scenario testing, two of the test techniques in the HTSM. It’s the motivation for pretty much all of the Quality Criteria in the HTSM, and it’s the motivation for the Customer element in the Project Environment. It’s the motivation for the HICCUPPS model for oracles based on consistency heuristics (http://www.developsense.com/articles/2005-01-TestingWithoutAMap.pdf). Heck, it’s pretty much everything that James and I talk about in terms of broadening testing skill. 🙂 Diversity is crucial, as Stefan suggests. Heuristics are crucial (and, since they’re context-sensitive, they’re also time-sensitive as Martin suggests). Dynamically managing your focus is crucial, lest you get too focused on one dimension of quality at the expense of the others. All of these are central to Rapid Testing.
Karl Weick put it very nicely: if you want to understand something complicated, complicate yourself.
(I think both of the Kaner quotes you cite are originally from James Bach—but they’re in the habit of crediting each other for inspiring their quotables.)
Cheers,
—Michael B.
I don’t do this consciously.
Maybe unconsciously and backwards, by looking at a product with all my senses in many directions, and when something catches my attention, I look carefully, and think about different types of users and situations, in order to find out how important this could be.
Seems difficult to be truly subjective for someone else, but well worth trying!
@Stefan:
I agree about the diversity, and even better would be a diversified team where each team-member tries to be as multidimensionally subjective as possible.
I am not saying that this multidimensional subjectivity would replace diversity; there is, and should be, room for both. They complement each other.
@Michael:
Thanks for the pointers to the RST and HTSM pages! (To any of you that has not read this, it is great reading!)
The RST methodology is a great inspiration and something I really like; it is testing as it should be… 🙂 So I am glad if my thoughts regarding multidimensional subjectivity is something that is aligned with those thoughts.
@Rikard:
I am not sure if I do this consciously, most truly this is an unconsciously act.
But isn’t this the case with a lot of things we do when we test? Many heuristics come to you in a split second, the moment you need them.
That’s why checklists aren’t the solution to the testing problem (replace all skilled testers with unskilled people carrying checklists), they would be too long and ineffective if they would cover all important areas.
And I am not sure if “truly subjective” is something achievable, and it is possibly not necessary. I guess that it is enough to be subjective for someone else, and that this “someone else” is a persona. Being empathic and drawing subjective conclusions based upon the persona or person is good enough (I am not sure what you mean by “truly subjective” and if that even exist…).
In relational psychoanalysis the psychoanalyst try to act like some person in the clients life, and this way try to really put herself in that persons situation and how this person experience the client. This means that it is the analyst’s subjective thoughts, based on another person, that is of help in the analytic process.
[…] Emilsson has written a good article on above quality […]
[…] Emilsson has written a good article on above quality […]
[…] Emilsson has written a good article on above quality […]