HICCUPPS F.C. Rikard Edgren
James Bach and Michael Bolton has a classic collection of consistency oracles; HICCUPPS(F): History, Image, Comparable Products, Claims, User Expectations, Product, Purpose, Standards and Statutes, Familiarity
It is a very good collection; not only helping you find out if something is a problem or not, but also the other way round: serving as testing inspiration.
At EuroSTAR 2009 tutorial Exploratory Testing Masterclass, Michael said that anyone coming up with additions would become famous.
As a tester this was of course an interesting challenge!
So over time, I now and then have thought of additions, but they haven’t been solid.
For example; V as in Violation of Quality Characteristics, could be seen as an inverted Familiar Problems or User Expectations.
You, as in the tester’s subjectivity can also be part of F and U.
But finally I found the important oracle missing from their list.
C as in Conversations.
You talk to a developer or other stakeholder, and collaborate to reach a decision.
Yes, during the discussion you might use HICCUPPS(F) oracles, but not necessarily. And it might not be the whole story, and not your oracle.
It might not be a 100% consistency heuristic, but is important enough to not leave out.
C as in Consistent with Conclusions from Collaborative Communication.
Outside the list we have the important No Oracle, where judgment is suspended, and noteworthy information is communicated, without deciding if it’s a problem or not.
But that doesn’t make for a nice soccer-sounding mnemonic, and is better left as a side-note to HICCUPPS F.C. heuristic.
(When double-checking, I noticed that they recently have added another one in the RST slides: Explainability – The system is consistent with my ability to explain it.
For now, I’ll put that one as a sub-part of Conversations; in dialogue, we weren’t able to explain the behaviour.)
History – the product should be consistent with previous versions
Image – consistency with the looks and behavior expected of the product
Comparable Products – consistency with competitors, references or other solutions to similar problems
Claims – consistent with written or oral statements about the product
User Expectations – consistent with what a reasonable user would expect
Product – consistent within itself
Purpose – consistent with the reasons for making/using the product
Standards and Statutes – consistent with standards or other authorities
Familiarity – inconsistent with problems seen before
Conversations – consistent with conclusions from dialogue
I can’t see ‘Conversations’ as a strong consistency oracle, Rikard (yet I still see conversations as a powerful activity).
As you say, conversation is a mean, that can use oracles to get a conclusion (or not).
If the conclusion of the conversation doesn’t borrow from the Oracles, it is because it comes from common Intuition — which I think does add a consistency oracle, “I”. Maybe I’m proposing a HICCUPPS FEI? (or “EPIC CUP FISH” if I dare rearrange them anew).
1) I would argue that “conversation” is an oracle no more or less than “documentation” is an oracle. Both are media in which Claims are expressed. (In class, I often point out that a claim might be made in a requirements document, a diagram, marketing material, or a hallway conversation.)
2) Explanation may not happen in terms of a conversation with someone else. If I can’t explain the behaviour to myself, I suspect that there’s a problem. (This doesn’t rule out the possibility that “conversation” could be my own internal dialogue, of course)
3) When we look carefully, the categories in this model (and in all our other models) are subject to Joel Spolsky’s Law of Leaky Abstractions. Our categories leak like crazy, allowing some ideas in and letting other ideas escape. In Rapid Testing lore, this is okay; categories that are too restrictive increase the chance that we’ll miss an important problem. Categories that are too permissive might do that too, of course, since they might let in more ideas than we can keep track of. So we address the issue by having plenty of categories, adding new ones as we recognize them, and winnowing the list every now and again.
4) Apropos of (3), we might notice that “History” is a special case of “Comparable Products”; that “Familiar Problems” ultimately evaluates to one of the other HICCUPPS heuristics; that “Standards” are probably either explicitly or implicitly claimed in “Claims”, and so forth. So there’s some overlap there so that the categories reinforce each other.
5) Apropos of all of the above, as always–and as you’ve done so very well, so many times–you’re absolutely welcome and encouraged to develop and extend our models and make them your own. Thank you for doing that!
—Michael B.
Thanks for comments!
They make me realize I shouldn’t call my list consistencies, and that I need to add something for intuition/subjectivity/feelings (and maybe that category includes Explainability; I am confused by product behavior)
I see HICCUPPS as a powerful training tool, when testers are learning how to recognize if there is a problem or not.
If Conversation isn’t on the list, some testers might forget to talk and learn more, they might make premature judgments.
In my experience conversations about test results sometimes generate new insight, e.g. the tester’s and developer’s perspective collectively decide if there is a problem or not.
So there is more to the conversation than claims (but there is not more to Documentation than Claims)
These leaky definitions work differently for different people, that’s a strength!
I used to like categories that were “true”, and would have dismissed Conversations as too different.
Nowadays I want things that are useful, that capture what is important.
I have taught this now and made up my mind (for now…)
HICCUPPS(F) stays as it is, a very good list of consistency oracles.
To Product, I add “self-explanatory, product says it has a problem” (abort/failure dialogs, log files)
After this, I say there are more ways to find out if there’s a problem or not:
* Conversations; learn more from someone else, or together.
* You: your feelings and subjectivity say a lot, and can be enough.
And we end with No Oracle, where you suspend judgment, and either wait until you now more, or report what you have, as noteworthy information (no decision if it’s a problem or not.)
If you want to lump all of these together, you get HICCUPPSF N.Y.C.
Explainability is out, because my experiences of it are negative. I have seen testers that don’t recognize a problem, because they can explain the behavior (mostly from a programmer perspective, e.g. “it is really difficult to avoid focus problems in a situation like this, so I won’t report this”)
It can be a useful oracle – “might be a problem, because I can’t explain it”, but it is too easy to think “if I can explain the behavior, it is not a problem”.
We have added (at the behest and demonstration of a student in class) a new one to the list: explainability.
The product should be consistent with your ability to explain it. Hence, if you see something in the product that you can’t explain, there may be a bug.
In class, a student saw an icon that didn’t make sense to him, and indeed it turned out to be the wrong icon. The closest thing to this is user expectations, but in this case it’s not the user perception that matters, or a simulation of user perception. It’s the tester trying to make sense of the product.
It actually echoes your notion of “conversation”, although I agree that conversation is a medium and not a consistency relationship.
I would call Explainability an indicator, just as feelings.
And I wonder about “indeed it turned out to be the wrong icon”:
How did you find out it was the wrong icon?
When you “knew”, you were probably using the true oracle.
[…] cục” mà mình đang nói đến là “HICCUPPS”. HICCUPPS là chữ viết tắt của “History – Image – Comparable products – Claims – User’s […]