thoughts from the test eye

This Wednesday I held a presentation at DevCon 2011 entitled Exploratory Test Design (slides)
I like this terminology a lot, because it encompasses the two things I want to see in software testing
* looking at a lot more information sources than requirements
* vary execution and look for many things

It was also a good presentation experience, because I could cherry pick my favorite heuristics, and talk freely about the ones I care most about, right now.
The audience was mainly programmers, and they are equally interested in exploratory testing and software potatoes!

But the reason I write this post is something Joakim Holm said at his talk, that should be told more often:
if you do a lot of testing, development goes faster

When testing provides continuous feedback, developers understand what is good, what is unreliable, and also what is important.
Continuous corrections towards the goal speed up the time to completion.
Testing isn’t a role with its own goal, we are there to help; and I think the most important help is speed, and the second one is quality (acceptable quality can be met in many ways, with testing it is faster.)

Someone might object and say “on the contrary, with all their phases and complaints, testing makes things much slower!” so I will rephrase:
good testing makes software projects go faster

During my first paternity leave I learned sourdough baking. During the second I couldn’t help writing an ambitious paper, or a small book, about people-oriented test design, about things beyond test design techniques, close to the exploratory testing tradition.

It can be downloaded here.

It contains collections of knowledge, and generalizations of my ten years of testing the same product suite. I think it can be useful for ambitious testers that want to find any problems that might be important.

It probably is too much, theoretical, irrelevant or condense for many of you, but if you want to give it a shot I recommend the following:

* Download The Little Black Book on Test Design
* Print as double-sided A5 Booklet
* Find a quiet, comfortable place
* Read and relate to your test reality

Comments are welcome, especially additions to the collection of one hundred and three test design heuristics.

How do you find out what is important, in your specific situation?

I think it is the essential problem for all activities with complexity.
I think it is impossible to do really good software testing without the ability to dismiss things as not important, and dig deeper for matters that are important (I believe it’s the same for cooking a meal, or raising a child.)
I don’t think you can sample appropriately unless you know a lot more than the requirements.
I think we need better understanding and models for this, if we are to persuade those that want to govern by number; they aren’t hitting the crucial things.

It doesn’t seem like pointing to intuition and subjectivity is enough, but I haven’t reached further than this:

Either you know what is important (when you see it), or you don’t.
So the more knowledge you have about things that matter, the better suited are you to find important quality-related information; some as planned, some by serendipity.
So the key is to learn a lot, from a variety of sources.

I obviously need help on this one…

I know of some testers who are pointed out by others to be Bug Magnets; people recognized for their ability to somehow draw bugs to them. Bug Magnets can be found in many workplaces and I bet that you know of someone that falls under this description. I have been appointed a Bug Magnet by some and it have made me thinking on what this phenomena boils down to.
Is it luck? Is it faith? Is it an ability that some are born with and some aren’t? Can you learn this ability? Can you improve it?
I have wondered about this for some time.

However, this summer I had a revelation when I watched an episode of the marvelous TV series “Homicide: Life on the Streets”:

Homicide: Life on the streets
Season 1 Episode 9 “Night of the dead living”
Det. Frank Pembleton and Det. Tim Bayliss

Bayliss [sits pensively. To Frank]: What are you looking at?

Pembleton is playing cat’s cradle.

Bayliss: You have something that you wanna say to me?

Pembleton: Adena Watson. So many unanswered questions.

Bayliss: And you’re saying that I’m not asking them.

Pembleton: I’m saying that you’re not answering them. [He peers at Tim through the cat's cradle.]

Bayliss [sighs]: What questions aren’t I answering, huh?

Pembleton [gets up and flips through a notebook, draws a picture]: Okay, these sixteen row houses on the north side of Kirk Avenue. Adena’s body was found outside the kitchen door in the red yard at 718 Kirk. Now the killer could have dropped her anywhere. Why not the common alley? Why not the yards at either end of the block? These three row houses are empty. One, two, three. The killer would have stood much less of a chance of being seen if he’d dumped her body in any one of these yards. Why would he risk bringing a little girl’s body inside a closed fence of an occupied house?
Maybe he wanted her body to be found immediately. Maybe he wanted to cast suspicion on the people in 718. Maybe he had some … perverse sense of remorse, some impulse to leave her body inside an enclosed yard to protect her from stray dogs.

Bayliss: These are *exactly* [taps the notebook] the questions that I have been trying to answer.

Pembleton: Well, you can try, but you never will.

Bayliss: Why?

Pembleton: You don’t think like a criminal. You don’t have a criminal’s mind.

Frank walks away. Tim grimaces in disbelief.

Bang! Suddenly it struck me. “You don’t think like a criminal. You don’t have a criminal’s mind”.
In order for a police to think of possible outcomes of a crime they have to be able to think as criminals, and put themselves in a criminal’s way of thinking.
If you don’t do this, you are trying to understand and explain the crime based on your logic.
Similarly, in order to “attract” bugs you have to wanna see problems; you have to identify problems that might bug several kinds of stakeholders; you have to put all your knowledge about the project in to consideration; and you have to be able to see the problems that matter. You are not trying to understand and explain the system by using your own logic, instead you are using several input sources to do this: logic, subjective thoughts, people’s skill levels, complexity of system, technology, etc.

So let me present my take on a definition of “Bug Magnet”:

Being a Bug Magnet = Able to foresee possible problems (or able to spot opportunities for things to go wrong), in context.

The most important thing here is the last two words “in context”. That is, even if you have all the bug taxonomies and oracles in the world to support you, you have to be able to understand what matters in this project.
Knowledge about “all common problems in .NET applications” can help you sometimes. Knowledge about “all common problems in .NET applications that developer X often produce”, is however much more useful.
Understanding what bugs our users is more helpful than knowing about what bugs users in general.
Knowing about problems with focus in Windows applications is one thing, finding these problems during testing is to be able to spot opportunities when they are presented to you in your context (see http://thetesteye.com/blog/2010/11/windows-focus/ ).

Now, back to the questions in the beginning of the post.

• Is it luck?
  – No, even if luck sometimes help. For others it is more of welcoming serendipity.
• Is it faith?
  – I don’t believe in faith.
• Is it an ability that some are born with and some aren’t?
  – Maybe. Some people might have a more developed talent, but I think that most people have this talent. Some are born with variations of narcissistic personality disorders and might have difficulties with this.
• Can you learn this ability?
  – I believe so.
• Can you improve it?
  – Yes. However, you might need to consider one or several dimensions to improve: Empathy, reasoning, attention for detail and seeing the whole, recognizing patterns of your own and other peoples mistakes, subjectivity,  general systems theory, context-driven testing, and more.
  Notice that these dimensions are not technical but rather comes from social sciences.

quantitative numbers in a world of qualitative feelings

I am not against measurements in general, they can surely be useful. I use length when building things, weight for baking, time for appointments etc.
I often use numbers for various things in my bug reports.

But metrics are something different; metrics are measurement plus value.
“Should have at least 80% code coverage on unit tests” is a typical example, where “peer reviewed and accepted” would give better results.
“2% better defect detection percentage since last release!” says less than conversations with support department.
“95% Pass on test cases” means nothing at all.

The measurements with value cannot judge what is important;
reality is impossible to aggregate;
metrics are dangerous.

There are many good software testers that advocate metrics. At a couple of occassions I have had the chance to have “the talk” with some I respect.
It boils down to the same thinking: metrics must have a lot of context in order to be useful, so much details that I believe you could throw away the numbers, or only have them as a footnote.
This is not done, because management demands numbers, that’s what they are used to base decisions on.
But for software and testing that might not be well applicable.
If kids want candy, it doesn’t mean you have to give it to them.
You give them better things, out of respect and care.

Testing provides information, so you should find out what is important to different people, and give them what they need, not necessarily what they want.
Metrics will hide importance, and also skew the development efforts.
Try to take them out, with surgical precision.

Then you are left with one final catch:
Qualitative information is hard to aggregate, trust is needed.

Story telling is an important part of testing. It is a part where you communicate and tell a compelling story what information you found at. Each story needs a scene in which it plays.

Some of you might have attended the weekend or weeknight test sessions, some might have attended classes in testing where they teach by painting you a scene in which you act. For those of you who are familiar with roleplaying games or story telling games, this is just like the story teller describing the scene and where you as a character act in it as well as interact with other characters. In order to learn the most and to be able to act the best of your abilities, the story teller need to paint the scene so vivid that you can imagine it yourself. The characters you interact with also need to be full of details, have an history and have agendas of their own.

When I took the class in Rapid Software Testing by James Bach, he had several exercises where he painted scenes that we interacted in. Some in the class were very unused to this, but James played the story well and coached them into acting. The best exercises were those that he had been practicing a long time that had lots of details and where you interacted with many different characters. Each character had a history of their own and he could tie them to individuals that he himself had interacted with in past projects. James laid out lots of traps that you were supposed to avoid or escape. He sometimes named what you did so that you can use it as an ability or heuristic in the future and more easily talk about what it was.

Some of you might be experienced with MUD:s, MUSH or similar. These are text based games that sometimes have a flare of role playing in them. As a Wizard (a programmer or game master) you sometimes had tools to you could utilize to make the scenes more vivid. You added *feelings* and smileys to your regular text messages to make a distinction between your moods. This has become a natural part of our language when chatting. Still, you can use it even more vividly when story telling in a chat.

How can you then apply your story telling skill to the test session? Here are a few ideas:

• Paint the scene well before the session, with all characters, the mission etc
• As a group of participants, what is your role? Are you a team? Do you know each other from before? How were you gathered? If there is a mix of skill and experience, how do you explain that? How do you intend to act it out? If you are experienced will you act in a different way so that the traps laid out will give a better learning experience for the in-experienced ones? It is good if this is defined before the session start.
• Put more details in the characters. Let them have names, titles and a brief history. It will make the decision making, interaction and role playing in the scene alot better.
• As a participant, when asking questions be clear to whom of the characters you talk to.
• As a session leader, when asked questions be clear on which character that answer.
• As a session leader, be clear when you want to give out general information about the scene, information about the characters in it or things that the characters notice as one “voice”.
• As a session leader, use the power of story telling such as “Meanwhile somewhere else … ” where you paint a scene to show things that happen outside the scope of the camera. This is a good trick to bring even more detail and power to the story being played out. Visualizing what you do with the camera such as zooming in or out can also paint a more interesting scene.
• As a session leader, do you want someone to act test lead so that they can practise leadership and organisational skills? If so, perhaps coach them into the story and scene a bit before hand.
• When chatting add feelings and moods to your regular chat message so that you can roleplay differences in what you say, as hidden messages between the lines.

I understand that there is a limited time issue both for preparation and when actually doing the session, but doing some of these things might help to some extent. Who knows, you might learn more?

This is a follow-up from Working with the testing debt – part 1 [1] and part 2 [2]. The reason for the clarification is that you so easily come up with a tip without a context or example.

Tip 3: Grow into a jelled team (read Peopleware [3] by Timothy Lister and Tom deMarco for inspiration). Peopleware identifies, among other things, things that you should NOT do in order for a group to grow into a jelled team. As a team it is so much easier to gain momentum in testing and especially so if you are jelled. Do you need to reconsider how you work as a team?

In one project I was test lead and tester. I had two fairly inexperienced testers assigned to me. I used a somewhat exploratory test approach to the planned tests, but they were far from organised and not well communicated. The tests consisted of a long list of one-liner test cases/test ideas. They were quite vague, thus enabling the tester to think for him-/herself. I had added some very specific test cases that were meaningless to run several times, but they were mixed in with the more vague ones. The testers thought it was meaningless to run the test cases several times, but I wanted them to do it anyway. Still, my intention was that they only use them as a guide. I did not communicate well and the testers did not feel like they were part of the test planning (which they weren’t) and thought I had set it up wrong. The mood in the team was foul. The testers had a hard time working with me for several years after that. Looking back I think they spent quite a lot of time that did not give any new information from what we had.

In another project I was assigned as tester together with my regular test group. We worked as consultants at the time. We did not know the project manager, test lead or any of the other testers that well. We were assigned test cases in different areas of the product suite. The test lead walked around and handed out test cases. There was no collaboration in the team. Sometimes when we talked over a coffee we noticed that we had done double work. Some tests were to be run in quite advanced configurations that took 90% of the time to setup. After a while we realised we spent 70-80% of the time setting up configurations, 10-15% of the time reporting bugs and the rest actually testing. By collaborating more we could have saved time to better switch tests around in the configuration that were needed between us. The team composition was 70% new consultants and 30% perm. employees.

And in yet another project the test team had worked together for quite some time. The whole team was involved in creating test missions/charters. Everyone tested and assisted in changing the test plan. When there was a major area that needs to gain some focused testing we assembled a group who generated test ideas, did some test design and did collaborative notetaking on the specific area. During the day the group debriefed several times and changed the test scope based on the feedback. Everyone was involved and felt great.

At one company we had assembled all testers into one big team (see [4] and [5] for setup) where we assisted all projects, the support organisation, the marketing organisation as well as taking on special missions from managers at different levels. When we started we had some unresolved conflicts, but as the team grew and as time went by the conflicts diminished. We worked hard together serving the organisation as best we could. As the team set forth we had the Black Team (inspired from Peopleware) in mind. We challenged the organisation to give us more to test, we could take on anything. We might have taken a bit too much pride in our work, but still I think that was needed after years of being looked upon as the nagging, complaining test team.

At the same company as above we sat in groups of four. When we got a new build we were armed to the teeth with test ideas, assigned areas for investigation and energy to shot down (not literarely) anything coming our way. During testing someone could shriek out in delight as a critical bug was found. The whole group would sometimes plung in and attack the same area from different angles to root out more issues, then continue on the track that they left earlier. We longed from the untouched (by testers mind) new features. We overwelmed the organisation with bugs, never having to consider that we might be the bottle neck at any time.

We all act differently depending on the context. With experience you hopefully learn from many of your mistakes. After you have been in a jelled team, you wish to be in the same situation again. You now know how good things could be and when you are not in that flow, you try to determine how it could be so. The book Peopleware is one of those things that might get you ideas on what you or people around you should stop doing  in order for your team to become jelled. Do not be afraid to tell management things that they that will stop you from growing. In some cases they might not know that they are doing it in a way that corrupts.

How does this affect the testing debt?

The benefits are obvious, a sentence to rise a cautioning finger about, but perhaps somewhat valid in this context. Part of the dilema is how you perceive the team that works well together. Do you see it is as a Clique (roughly described as an arrogant group) or as a Jelled Team? For my description I emphasises on a team that see itself as a jelled test team. When I say to increase the testing debt I mean that your backpack is getting heavier. Each time you start a new task, new project or new assignment you take the content of your backpack into consideration. Being a jelled team enables you to move more smoothly and become more flexible, thus able to become quicker and hopefully deliver better result.

We should also consider that there are differences between a team that is jelled consisting of various roles (such as a cross-functional team) and a team with only testers. Both constallations have different issues and both will have an ever growing testing debt. Some tips would apply to one constallation and not the other, naturally.

Kay Johansen and Anthony Perkins identified a few interesting ideas in their paper Establishing an Agile Testing Team: Our Four Favorite “Mistakes” [6] that I think is relevant to building a jelled testing team. We also have a paper by Elizabeth Hendriksson on Agile Testing [7] that identifies things you should not do as a tester if you intend to be agile. They are most relevant when considering what role testing should have in a cross-functional team or what your attitude is as a test team. Brian Marrick has made a compilation of articles [8] on Agile Testing, somewhat old ones but still valid as a way to consider when trying to become more effective and build a jelled test team. Marrick has also created the article Classic Testing Mistakes [9] that is valid still and should be considered when trying to identify why your team is not jelled yet. I’ve also written two blog posts about growing test teams: Uncertain team composition [10] and Progress [11]. Both tries to focus on things you should not do in order to get a jelled test team.

Some specific things that I see a jelled test team do, that decrease the testing debt, is:

• Build on each others test ideas. With this I mean that we are triggered on each others test ideas to create new ones until we have no further ideas.
• Easy of changing direction. I often promote the concept that the test team or the team in general need to be flexible, thus the naming of The Flexible Test Team. If the team works well and trust each other it will be easier to accept change.
• Collaborative notetaking. In a team where you want to work together and where you enjoy cooperation it is the teams effort that counts, not the individual. This is also true for taking session notes. This is no best practice (as none exists), but a practise that is good in some situations. That the team have the ability to do this is a bonus.

With an unjelled team, a group of people that do not collaborate as well as they could, thus increasing the testing debt, I instead see:

• Stacking of test ideas. This means that each individual identify test ideas and they are summed up, with duplicate ideas removed. An unjelled team is more inclined to do this than a regular jelled team.
• Double work and duplicate testing. This might be the result of bad leadership and test leading, but based on my experience I think it is more common than in a jelled team.

What things do you consider for a jelled test team or a jelled team that has mixed roles but with testers in it? What have I missed as you see it?

References

[1] Working with the testing debt – part 1 - http://thetesteye.com/blog/2011/05/working-with-the-testing-debt-part-1/

[2] Working with the testing debt – part 2 - http://thetesteye.com/blog/2011/06/working-with-the-testing-debt-part-2/

[3] Peopleware - http://www.amazon.com/Peopleware-Productive-Projects-Teams-Second/dp/0932633439/ref=sr_1_1?ie=UTF8&qid=1310020112&sr=8-1

[4] Flexible testing team - http://thetesteye.com/blog/2008/05/the-flexible-testing-team/

[5] Resource planning in a flexible test team- http://thetesteye.com/blog/2008/05/resource-planning-in-a-flexible-test-team/

[6] Establishing an Agile Testing Team: Our Four Favorite “Mistakes” by Kay Johansen, Anthony Perkins - http://agile2004.agilealliance.org/participate/examples/EstablisingAnAgileTeam.pdf

[7] Agile Testing by Elizabeth Hendriksson - http://testobsessed.com/wp-content/uploads/2011/04/AgileTestingOverview.pdf

[8] Agile Testing by Brian Marrick –  http://www.exampler.com/testing-com/agile/

[9] Classic Testing Mistakes by Brian Marrick - http://www.exampler.com/testing-com/writings/classic/mistakes.htm

[10] Growing test teams: Uncertain team composition - http://thetesteye.com/blog/2010/04/growing-test-teams-uncertain-team-composition/

[11] Growing test teams: Progress - http://thetesteye.com/blog/2009/10/growing-test-teams-progress/

I recently acquired the best product in the world, for me.
It is a Victorinox serrated multi-purpose chef’s knife that I use to slice bread.
In 2000, chefs around the world voted it as the best knife in the world.
I want to go one step further, and name it the best product in the world, all categories.
Some people say that if you go to a restaurant, and they don’t have this knife anywhere, you might as well leave at once.

When I use the knife it has an incomparable feeling when it cuts through the hardest crust of a sourdough bread. There is an ease in its operation that brings well-being.
It might have drawbacks I haven’t noticed yet, but with this 100% Charisma I don’t really care, I could slice bread for days.

So what kind of charisma does your product have?
Can you test for this and make suggestions for improvements?

In testing text books you can read that compatibility testing should be performed after the functionality testing and bug fixing is completed. I guess the reason is that you don’t want to mix these categories, but hey, what a waste of resources.
My suggestion is to perform the compatibility testing at the same time as you are doing your other testing; when problems arise, trust that you will deal with them.
In my classification, compatibility testing involves hardware, operating system, application, configuration, backward/forward compatibility, sustainability and standards conformance.
Here follows some lightweight methods to tackle these areas.

Basic Configuration Matrix

Basic Configuration Matrix is a short list of platform configurations that will spot most of the platform bugs that could exist in your currently supported configuration matrix.
The simplest example is to use one configuration with the oldest supported operating system, oldest browser etc; and one configuration with the newest of all related software. A more advanced example could use several configurations that use different languages, Application Servers, authentication methods et.al.
Often it will take quite some time to run most tests on BCM; so alternate between the configurations while testing your product. Do variations on configurations when appropriate.

Error-prone Machine

Another trick is to setup machines so they have a high chance of stumbling on compatibility issues. You can vary this on your BCM, your personal machine or whatever is suitable. The idea is to get some compatibility testing almost for free.
Examples on Windows include: run as Restricted User, use Large DPI setting, use German Regional Settings, install support for all of the worlds characters, non-English language in Internet Browsers, move system and user Temp folder, activate IE ‘display a notification about every script error’, move Task Bar to the left side of the screen, Windows Classic Theme & Color Scheme, use User Access Control, use an HTTP proxy, use Data Execution Prevention, install new versions as they come, e.g. latest hotfixes, MDAC etc, never install software on default location, run with 2 screens, on a 64-bit system, use different browsers, turn off virtual memory swapping, , install Google/Yahoo toolbar, run Energy Save Program, pull out the network cord every time you leave the computer; and put it back in when you return, turn on the sound!

Technology Knowledge

If you know a lot about the environment the software operates in, you know which things will happen in reality, which settings that usually are altered, and how it is commonly operated.
The lightweight method is to use this knowledge and make sure you test the most important things.

Letting Others Do the Testing

Many compatibility problems happen on basic usage, which indicates that you can let others do a big part of the compatibility testing: developers can use different machines and graphics cards, Beta testing can be done in customers’ production-like environment. If the product is free of charge, you might even get away with addressing problems after your users encounter them (but make sure you have an easy and compelling way for them to do this reporting.)
Crowd-testing could be a way, but so far the payment models from the testers’ perspective are not ethically defensible, to me.

Reference Environments

To quickly investigate if you are experiencing a compatibility issue, it is handy to have reference environments available. It could be someone else’s, a virtual machine, a quickly cloned image, your own machine etc.
Personally I prefer having a physical machine that is running similar things, but on a different OS, different language and/or earlier version. The last years I have had three machines and three monitors, and by switching, I get a lot of compatibility testing done at the same time as testing new features. When I check things on an older version, I can save documents and use them for next tests.

Backward/Forward Compatibility

Backward compatibility is easiest done if you can use real customers most complicated files/data/documents. Use these as you test any functionality (Background Complexity Heuristic).
Occasionally communicate between different versions.
Forward compatibility should be designed in the architecture, as a tester you can point this out.

Sustainability

Have conversations around the question: Is the product compatible with the environment? Have we considered energy efficiency, switch-offs, power-saving modes, support work from home and the likes?

Standards

A lightweight method for standards conformance is to identify which ones are applicable, and ask the experts if they understand it, and successfully have managed to adapt it to the new context.
Let’s finish with a non-lightweight method: you can become the standards expert.

I used to be a bit skeptic towards the word “heuristic”. It seemed needlessly advanced, and things are possible to explain in other words.
But when I read Gigerenzer’s Gut Feelings about how to catch a flying ball, it all came together.
For software testing, which can’t be complete, is dependent on many factors, with a product to be used with different needs and feelings; techniques are not appropriate. It is about skill, and human skill is very good to describe with a variety of heuristics.

When blogging about some heuristics I think are un-published and worth knowing about, I’ll try to do two at a time; more bang for the bucks.
With English as second language it is difficult to give them good names, so feel free to suggest better names! (and content…)
And remember, heuristics are not rules; they are more like rules of thumb, that might be useful, in specific situations.

No Flourishes Heuristic

At many times you can design and execute straightforward tests, without garnish, fancy tools, and incomprehensible details.
Try this when you have the chance!
See if perceived performance, manually clocked, can give good information.
Use common options instead of combinatorial.
Look at the GUI instead of automating tests.
Try to do something valuable instead of covering all paths in last years use case.
Write your basic test strategies in plain English, so everyone can review them.
Use what you have, and look for what’s important.

New Connections Heuristic

How do you “discover” what is important?
I think it often is about combining different knowledge in new ways.
So you need to know a lot about the product and its context, and look for connections between the wide diversity of knowledge you have.
When reading, talking, thinking, you sometimes instantly think “but this could have big impact if one is trying to do that!”
Or during test execution, when you suddenly get an impulse that you have to add some other things to the stew.
This might seem unstructured, and dependant on chance, but that is OK. If software testing is a sampling problem, we need different ways of discovering what is important.