A Hot Model That Excites Me Torbjörn Ryber
When I meet testers I often ask what test design techniques they use on a regular basis. I am really curious which of the techniques that I know of that they use and I am always looking for new material. I have taught test design classes to maybe a thousand people over the years and I keep asking for THEIR examples of applying techniques after the class is over. So far I cannot remember having received any specifics at all. So I am starting to wonder..
There is so much talk about test ideas, heuristics and lists and although these are wonderful new ideas, I am afraid that we are forgetting a core skill that can be very rewarding. One of my favorite techniques is state modeling. The ISTQB foundation class shows you a table or a graph and you need to identify what series of events are possible. This is much like the puzzles for children where they have to help the dog find the bone in the maze. Well, no surprise, this is just their way of simplifying something beautiful to total pointlessness. So let me tell you how it can be done better.
A project I am currently is an application for administration of scientific reports. A researcher sends their report to an editorial board which then assigns an editor and a numbers of referees. Then the evaluation process starts and reports come in. It all ends by a report getting refused or accepted. Now this is a school book case for doing state graph modeling. I would not find it strange of the developer had already created a number of such models for their own benefit, but as usual, there were none. Now I started to get excited. A really cool and useful model was to be created and I would be the father of it! Days like this are what make me happy!
Step 1) I read all documents I could find. I questioned the business users and got a pretty good picture of how they worked. Then I went to the developers and started asking questions about what statuses the manuscripts had at different points in the process.
Step 2) I created a state-chart and went back to the developers. The response I got was that they wished they had done that themselves before and they instantly saw a numbers of changes that would make their programming easier.
Image: The state graph as it is right now
Step 3) Creating a detailed state table. One thing I really like with this technique is that is very close to actual design of the code. The manuscript is in one state and then something happens that put it in another state and also a number of parameters should be updated. The manuscript is new, state 0, the admin selects editor and referee and presses button send mail. Now status should be changed to 1, e-mails sent to all selected referees and editor and a notification sent to the author the evaluation has started. When in state 1 and additional referees are selected, send mail notifies only the new people etc. This turned out to be one of the most important parts of my test design and the group solved a lot of issues together. So before I had executed any test cases at all I have already helped making design changes and found some serious omissions. I keep on asking questions like “What if this happens when I am in that state”.
Image: small part of the state table.
Step 4) Test execution. I listed a number of test ideas and ran each idea in a session inspired by SBTM. Logging was done in Word since this supports adding screenshots. I did start with the SBTM notepad format that can be analyzed but soon realized that there were so few logs and only one tester, that the benefit of logging text and pictures in the same place was better for me. So my form of exploratory testing is to have test ideas and test design models. The only scripted test cases we created were for the demo where the goal was to show that it worked. Needless to say we got some questions during the demo and deviated from the script…yeah it would have been good to test it some more before the demo.
Step 5) Continuing the project we realized that some additional functionality needed to be added to make it work smoothly. More specifically we added a new status 6 – published. This helped us a lot in selecting what manuscripts to show on different screens. The state model made it very easy to see the effects of our changes.
To summarize this: state models help finding bugs before executing test cases, they help clarifying the design, they guide me during test execution with overview and detail on a couple of pages and they are a great support when evaluating changes to the application. C’mon, how can you not be Excited by this Hot Super Model!
“So my form of exploratory testing is to have test ideas and test design models.”
I do exactly the same. But my application isn’t the best suited for a state model, so most of the model is inside my head, and some is in mind maps, and for some functionality I can re-use models made by others.
“I keep asking for THEIR examples of applying techniques after the class is over. So far I cannot remember having received any specifics at all. So I am starting to wonder..”
My experience is that classic test design techniques like decision tables, state models, equivalence partitions aren’t explicitly used for either of these two reasons:
1) the result is so easy to grasp that you do them super-fast in your head, and just jot down or execute the test ideas/test cases without formalizing according to a technique (e.g: equivalence partition is used sub-conciously all the time by all testers)
2) the result is so complicated that you’d want to select sub-sets of the tests anyway; and to do this you can use any type of explicit or impicit model; using a technique might be good, but other ways (a list, an image) can be tweaked more easily to fit the specific situation.
However, the more you know about these techniques, the better tests you can create; no matter if you’re using the exact technique or not.
I am excited about your model! 🙂
It reminds me of when I was a team leader of an interaction design team when we developed a model of a very complex design issue. This model was so useful during early design discussions that I really doubt that we ever had come near a solution in time without it. We tested different scenarios against the model all the time and found new problems each time. Very fruitful!
And I also remember a state model that we developed when I was test lead that displayed a model of states between our different sub-products. At the time we did it, it really revealed some important issues and it also made us aware of what important things we should look out for that could fall between the chairs.
Both examples are from the same application that Rikard described, which was not best suited for state models according to him… 😉
The thing as I see it is that you can create a model on any appropriate level in order to clarify – a model that is easy to communicate and discuss.
I must admit that I do this too seldom, but now when you reminded me of this I need to get back to it as soon as possible! Thank you for sharing this!
Just like test ideas is a powerful way of communicating what you intend to test in early test specs, so is state-modelling to better describe complexity and find new ways of attacking the system.
One funny thing that I’ve seen is when someone creates a really good looking model it can lure you into thinking that the system is not complex, just because it looks so smooth.
Either way, we should promote all ways of communicating around the system and the risks we see.
Rikard: I would rephrase the sentence to “YOU BELIEVE the result is so easy to grasp that you do them super-fast in your head and just jot down or execute the test ideas/test cases without formalizing according to a technique THEREBY MISSING THE OPPORTUNITY TO FIND OUT IF YOU ARE CORRECT OR NOT”
And the reasoning that it is too complex to make a model is flawed. It is even MORE important to model if something is very complex.
Martin: What happened in this particular project was that the design actually changed to an easier one! I still believ that Simple enginerring is great engineering. Or as Weinberg would Say ” There are no hard problems, only hard solutions”
And there is much more to the model above that I decided not to go public with that has been really useful.
Tobbe, I also think that models are very important for complex problems.
But I don’t think that STATE MODELS always are the best way to do it.
The classic design techniques are very good to know about, and you learn from them, but for real situations they are just a tiny part of the testing problem and solution.
“THEREBY MISSING THE OPPORTUNITY TO FIND OUT IF YOU ARE CORRECT OR NOT”
Experienced testers that don’t have unlimited time take these shortcuts all the time.
And personally, I mostly find it more fruitful to get feedback on test ideas from a colleague than to formalize and find the so-called “TRUTH”.
But of course, doing both can be the best sometimes.
I hope I have not said that state-models are always the best way to do it but I very often find them useful for som e part of the applications I test. Remember I claim to be a context-driven tester and will not say I have any “best models”. Most important is that I always find something interesting when creating the models, one part is a better understanding. Having said that I admit that at the time I wrote my book I was very focused on the mathematical/logical models that I had used succesfully at the bank I worked at. Test ideas are great but just like the state-models they are only part of the truth.
The only thing I know about my models are that they are always wrong.
“All models are wrong but some are useful” – George Box
http://en.wikiquote.org/wiki/George_E._P._Box
I found some more interesting info regarding models here http://www.idiagram.com/ideas/models.html
And i got a comment in Swedish regarding state-models not beeing very good for linear problems and my answer on that is that state-models are excellent to use for non-linear processes. Compare this to flow models like activity diagrams which are really bad for any complex processes. That is one reason I think step-by-step use cases and test cases are not a very good way of describing more than simple examples of flows.
We really need to put together a Peer Conference on test design within the next year. Maybe november/december or january next year since some of us will be attending EuroSTAR as well.
Swedish Television has a documentary (in Swedish) available at http://svtplay.se/v/2054577/dokumentarfilm/tecken_pa_hjarnan
It is called “Signs in the mind” and consists of interviews with people about how they visually see things like the alphabet (A-Ö with diminishing size), numbers (different color for each number) and time (a circle with Christmas Eve at the top).
It is fascinating how different models people have!
Personally, I don’t have a visual model I’m using for any of these things (but I still know the order of the letters in the alphabet); and I guess that’s reflected on the few visual models I use at work.
It’s also a very good subject for discussions among friends:
“What does your year look like”?