thoughts from the test eye

I go more lightweight than that.
First I saw of my toothbrush and then…

When a system is beeing developed I try to catch an intended future user, put them down in front of the application and ask the the simple questions:

1) How do you perform this specific function?
2) What do you need to do it?
3) One of my favorite testing heuristics: Do you understand what just happened?

It is scary how often those questions have not already been asked before starting to build the system. Here´s a 20 minute conversation that took place about a year ago.

Tobbe: So tell me what is the first thing you do when you find out that your client has made an error when regeitering info on the web.
User: Well we look at the errors and then contact them to ask for clarification.
Tobbe: So how do you cantact them
User: By phone usually.
Tobbe: But on the screen you are currently looking at there is no information regarding phone number?
User: that is not good, we need easy access to that info
Tobbe: i cannot find that info anywhere in the application. What next?
User: we note down what we talk about
Tobbe: Where do you enter those notes?
User: well, isn´t there a field here for that, no that was for a future release.
Tobbe: So you are telling me that the two most important things you need to work are not there.
User: I guess we forgot that when we said the design was OK.

I do the same thing myself after I have tried to understand the requrements but I always try to involve the users as well.

I also like to add to your list above:
1. Advanced User Potential: can an expericed user work (more)efficiently in the application
2. Stops you from making serious mistakes
3. A user enjoys using the system: sometimes the most important issue!

“ask the following questions to someone with many hours of using the product (probably a real user or a manual system tester)”

Someone with many hours of using the product is probably not the best person to assess many of these usability questions.
– Anything that was initially confusing has probably already been understood by this user.
– Anything cumbersome has been trained-in
– Anything not initially memorable has long since been memorized

It seems better to me to use someone who specifically has NOT spent many hours on the product.

Tobbe, great tips and story
In that (and an awful many) situations, I guess my list is overkill.
Maybe these sub-categories are better suited when the basic scenarios already are thought through, intended (first time) users have been involved, and you have a system you think is usable.
My intention was to cover two of your additions:
1. Operability
2. Errors
3. very important, but not part of Usability in my classification scheme (this other category is still “secret”, ask Emilsson about it…)

Joe; diversity is of course the best, but from a testing perspective I would pick an “advanced” user first also for your examples:
1) Those issues that still are remembered are probably the most important ones.
2) The person with knowledge often has people around her that asks questions, or complains about usability issues, or hear about problems at customers. These can be aggregated and transformed to a compelling usability bug, e.g. actions that “everyone forgets about” can be better explained, or available at another place in the User Interface.
Yes, this is subjective, but in a trustworthy way.

Traditionally, I believe “someone who specifically has NOT spent many hours on the product” are used for the majority of usability testing; as a system tester I like to do the opposite.

It depends on the context is as always the right answer.

An experienced user is probably not very good at finding learnability issues but excellent at finding problems with advanced usage or distilling most important issues regarding help, minimalism and errors.

If my application is a web interface for selling books the first-time user will spot problems with Intuitiveness. Can a new user easily buy a book?

Regarding localisation it might be better to set up a test lab or to have a really good production log.

I don´t mind having your long list as inspiration for usability testing but if the goal is to make a point that testers can and should be aware of usability issues. I prefer a shorter list. Five bullets – because this is reasonable to remember.

I find that I many times get the comment “everybody can have opinions about usability, it is subjective” So I find it importont to be able to explain the theories behind usability issues. My comments may be somewhat subjective but I am fairly well educated in the area so they are at least educated subjective comments. And I never tell them to fix the issues, I explain the problems that they may cause and ask what users and developers think about it. Usually they get fixed in the end…

“Five bullets – because this is reasonable to remember.”
This is interesting, because I have never felt the need to memorize list or mnemonics.
I wonder if it’s person-dependent, or situation-dependent (I have tested products from the same company in 10 years)
Sometimes I look at lists to get inspiration or learn things, but when actually testing I can see any deviations to Usability or other non-functional categories without a memorized list, the concepts are stored in the back of my head.
When I see an issue and report it, I can afterwords put it in a HICCUPPS(F) category, but I have no need for the memorized list.

I could argue now that since you see any deviations without a memorized list but find issues that fit to the list you wrote above you have actually not only memorized the list but are also using it without reflecting about that. If you have tested the same product for ten years I definetely say that you have in your long term memory loads of information that you use wothout reflecting about it.

I think that we are partly talking about different things. You are talking about a lightweight process to test usability for an experienced tester (?). I was talking about how a(ny) tester can start testing usability issues quickly.

When I teach thing, talk at conferences etc I try to limit myself to maximum five key points. If you look at many famous models like Myers_briggs personality indicator and variations of that – there are almost always four main types/factors and then they may be combined to make a finer grid.

Another great eample is Weinbergs model MOI(J) where he simplifies thing by saying that the four things that drive people are Motivation, Organisation and Information. You can affect that with the same three factors and sometimes by a Jiggle. I find it beautifully simple and use it very often.

I could go on about this some more but the point I am trying to make is that the easy models often stick whereas the more complex models do not.

When you test usability have you ever tried to focus on only 1-3 questions in one session and then compared this with testing every question at once. OFAT/MFAT…

Yes, I have tons of information that I use without reflecting about it; and I think many testers are in the same situation. And I believe that testers are most effective when they have a combination of “general information” and “product-specific information”.
But I wouldn’t say that the list is memorized, rather I have a few of them memorized, and a lot of fragments that somehow match this generalized list.

This post was not specifically aimed at experienced testers in the same situation like me; rather I was hoping that a word or a question could trigger a thought that the reader would learn something from.
You will probably focus on a few type of usability issues, but I think it’s better to create your own short list from a long list, than hoping that 5 easy to remember phrases from someone else, can capture the important things in your situation.

After compiling this list, I feel that I have more and better words for describing usability, and I have enhanced my sensitivity for some types that I wasn’t explicitly aware of before, e.g. Affordance.

I am starting to suspect that posts like these aren’t very productive for the majority of readers, but you know, someday someone will find the page and it will match what they need.

Hi,
I agree with Richard about having a longer list of heuristics is more sufficient for testers so a subset of it can be applied to their specific context. I really like those kind of lists where I can pick the cherrys for my cake that I am baking at the moment.

I also think it is important for testers to know these kind of basic usability rules. Projects rarely afford to have usability experts around that much, and having a tester that can point out those things could actually motivate getting one involved if needed.

Torbjörn, the short lists ARE sufficient in teaching and is easier to learn. But how often do you actually need to learn certain heuristics. I see heuristics as a setting of the current mindset that are brought out when needed. So if I get a list like this, I just read it through quickly, and then I might archive it for reference. I dont learn it, but know where to look for it.

Thank you for a nice post!
Sigge

Btw, your wordpress theme does not work very well with safari:
http://dl.dropbox.com/u/110771/safari.png

compared to firefox:
http://dl.dropbox.com/u/110771/firefox.png

Sigge: Are you seriously arguing that you think it is not an advantage to have a large amount of knowledge in your head when testing? This knowedge can be testing theory including heuristics and domain knowledge.

E.g., I am testing an application that seems to in really bad shape and I recall the Dead-Horse Heuristic http://www.developsense.com/blog/2009/09/when-do-we-stop-test/ Your method would then be to take a look in your archive and search for “crappy release” to see if you can find a heuristic that tells you what to do? Or read the information beforehand…

Not knowing requires that you remember to look at all, where to look and that you find that specific piece of information that you (don´t) know you are looking for.

And I have never argued that the lists are not great to have as inspiration, quite the opposite! And each time I read through them I find more cherries that I then store in my long-term memory! 🙂

Torbjörn: Thank you for pointing out my unintended use of the word learn. What I actually meant was that you rarely memorize these lists on the exact wording of it. Usually a quick read-trough and a little thinking in the context is what is needed to grasp the overall meaning of the bullets in the list. That is what is learned, while memorizing is easier on shorter lists. Of course the amount of testing theory and knowledge is important, I am not saying anything against that.

Actually, I rarely try to refer to a specific heuristic and its meaning, which is why I dont need to memorize it either. Such references is in my perspective not very important in most contexts. Or do you have examples on when you have actually referred to the Dead-horse or similar to a client or project stakeholder?

Thanks for the information.. it was useful

Great stuff have already been said in the post and previous comments, but I just want to add my thoughts on this.

The key thing, as I see it, is when interviewing “someone with many hours of using the product” you should try to avoid interviewing super-users. It’s like Joe said, they know too much of the system. And not surprisingly, in my experience they have often been involved in designing the system; and/or been involved in the requirement phase; and/or been involved in early user testing. Therefore they might be biased; or not affected by certain usability issues they don’t notice; or already have been involved in deciding them as not important.

And you should also try to ask open questions when interviewing someone if you are looking for information that you didn’t know about before.
Most preferably you should observe the user doing user specific tasks while trying to get the user to describe its intentions and ask open questions about their experience. You could still do this (in nearly the same time) and use the category list above as a reminder of important things to catch.

Of course testers have a lot of useful information about usability issues. It is just a matter of being important enough in order to have impact and seen upon as trustworthy. As with many design issues, there are as many opinions as there are people involved. Having real users (5-7) as a basis is often a good decision support; having one real user might be sufficient; having one system tester might be a waste of money.

I remember a project where I reported around 10 serious usability issues, and the bugs were reinforced with credible scenarios and likely effects.
None of these were fixed before going live. Instead they took away the old system and rolled out the new system to each department around the country. It took the users about two days to report the same issues and they refused to use the system until the bugs were fixed. The release was delayed with a couple of weeks and the departments stood still during this time.
In retrospect it is obvious that I was right, but the decision makers of the project had already interviewed real users, involved them in the requirement phase, and had let them test the system several times before the release. The problem was that these users were “super users” and they had been involved from the start; so since they hadn’t complained on these issues there were no issues. And who could blame the decision makers? They had done “everything right according to the book” and included real users right from the start to the bitter end. Why would they consider my thoughts? I had never worked in the production and had never met any real user.

Nice story, Henrik. A really good example of when the tester finds usability issues while “verifying the funtionality” (which I assume wsa the case)
And maybe because they hadn’t expected and planned for usability issues, they were reluctant to address issues found by a tester?
But it was fresh, critical eyes they needed.

Could it be that software people have learned that developers make mistakes and testing is needed, but still think that a deliberate design with user input will be perfect??

Sigge; in which requirements document did you read that we support Safari?
Just kidding, thanks for the input; we have a lot of alignment issues on some browser versions on some systems, but your example might be the worst.