Observation and interpretation by proxies Henrik Emilsson
If you haven’t done it before, have a look at the Software Quality Characteristics that we published last year: TheTestEye – Software Quality Characteristics
You can probably imagine ways of testing for all of these quality characteristics yourself, and you might even come up with good oracles that can assist you in the interpretation of the test results. However, the Charisma characteristics might be the most subjective ones; and it might also be a case of where it is really important to find out, for each quality characteristic respectively, which stakeholder whose values matters the most.
So how do you test for a product’s Charisma?
Charisma. Does the product have “it”?
- Satisfaction: how does it feel after using the product?
- Professionalism: does the product have the appropriate flair of professionalism and feel fit for purpose?
- Attractiveness: are all types of aspects of the product “good-looking”?
- Curiosity: will users get interested and try out what they can do with the product?
- Entrancement: do users get hooked, have fun, in a flow, and fully engaged when using the product?
- Hype: does the product use too much or too little of the latest and greatest technologies/ideas?
- Expectancy: the product exceeds expectations and meets the needs you didn’t know you had.
- Attitude: do the product and its information have the right attitude and speak to you with the right language and style?
- Directness: are (first) impressions impressive?
- Story: are there compelling stories about the product’s inception, construction or usage?
All of these are somewhat intuitive and I guess that you, consciously or unconsciously, test for these everyday in your project. You can also focus on testing these in the same fashion as when testing for other quality characteristics.
However, if (some of) these software quality characteristics really matter in your project and are core values for your product, you better question your own capability of being able to test this.
By question yourself; you might realize that your oracles aren’t powerful enough in order to really being able to answer the question “Is there a problem here”.
You might be too biased; or you might not be able to really understand what the stakeholders need or value. Or you might simply be too unimportant in order to even “have opinions”…
One way of addressing this is by letting a proxy user act as observer and interpreter of the test result. This can be a good approach when testing for software quality characteristics where you suspect that your own oracles won’t be enough in order to understand what is important. Well this isn’t news for many of you, since user testing has been used successfully for years. But my point is that you can spend some extra thoughts on which you select as your proxy oracle – with a mission to test for those selected Charisma values.
You can do this kind of testing by letting a proxy sit next to you while you test; or by letting users test the program themselves and you interviewing them afterwards. Another way can be to test or demo in front of an audience of proxies that observe and interpret the result as you go.
The tests could be designed as open questions that you would have asked the program and watched it respond, but instead you ask the questions to the proxies and letting them answer in the way they interpret the result.
Sounds interesting, and well-spent time when charisma is essential.
Sort of dynamic walkthrough?
I don’t think it is common that testers actively look at charisma, and when they do, problems are probably reported informally (nothing wrong with that), if at all.
Dynamic walkthrough is a good word!
I do think that testers look at charisma, but they might not have thought of it in these kind of wordings. My experience is that a lot of people, including testers, have opinions on design; opinions that aren’t always appreciated by the designers (they get a lot of feedback they think is just “one man’s opinion”).
And in many cases, the outcome of the design becomes charisma quality characteristics. So this way it might be easier to talk about these issues by referring to our poster?
It also makes it easier now to discuss the implications with respect to the quality characteristic since you now have words to describe the issue with, e.g. “I think that the Attitude of the product is wrong, the users aren’t used to social media language…” or “The GUI does not have the flair of Professionalism I would expect from a statistical tool, it looks too web-ish and does not have a serious look and feel.”
I definitely look at charisma. I meet the customers, I talk to the designers and the Interaction designers to try to understand what the product is all about.
When I then test I have a feel for how the functionality should be presented, and it’s important because a large part of what we build is presentation.
Even so, my “bug reports” on the subject is more suggestions for the designer on what to have another look at. There might be a bug there in the sence of miscommunication between the concept and the ready product.