thoughts from the test eye

Jerry Weinberg made an excellent comment on my previous article Turning the tide of bad testing [1] where he wanted more examples/experience on the tips. It is sometimes a bit too easy just to come up with a tip that lacks context or explains how you used the specific tip in a situation and where it worked for you. There are no best practices, just good ones that might work in certain contexts, still you might get ideas from it and make something that works for you.

Tip 1: Exploratory test perspective instead of script-based test perspective. This can be roughly summarized as more freedom to the testers, but at the same time adding accountability to what they do (See A tutorial in exploratory testing [2] by Cem Kaner for an excellent comparison). And most importantly, the intelligence is NOT in the test script, but in the tester. The view on the tester affects so many things. For instance, where you work … can any tester be replaced by anyone in the organisation? This means that your skill and experience as a tester is not really that important? If this is the case, you need to talk to those who support this view and show what you can do as a tester that makes his/her own decisions. Most organisations do not want robots or non-thinking testers who are affecting the release.

In one organisation I worked in, there was a typical scripted test approach initially. There were domain experts, test leads and testers among other roles. The domain expert and test leads wrote test specifications and planned the tests in a test matrix to be executed. The test matrix was created early and it was not changed over time. Then testers execute the planned tests. Management looked at progress by test cases, how many was executed since last report.

At one time I was assigned to run 9 test cases. I tested in the vicinity of all of them, but I did not execute any of them. I found a lot of showstopper bugs and a few new areas totally uncovered that seemed risky. To the project and to the test lead this meant no progress [3] because no test cases were finished. The test matrix had been approved and we were not to change the scope of testing, so the new risky areas were left untouched. A tester did not seem to have any authority to speak up or address changes in scope. As I saw it, we were viewed as executors of test scripts. The real brain work had been done before us. During this time, management appointed some new temporary personnel to the team that had little domain knowledge and no testing expertise. They did not think any experience or skill was needed. Just because it was so short time the new personnel did not get up to speed with the testing tasks. Some of the testers said it just weighted them down with extra hands. When we executed test scripts we were usually assigned a few and then ran them over a few days, cooperation between testers was not all too clear.

After this assignment I was able to be test lead myself and got all previous testers allocated to me. I started to introduce the exploratory test approach by letting the testers have more freedom, but at the same time being responsible for what they did. We started using test sessions as well as scripted testing as a transition period. We adapted the test matrix over time based on new risks. Temporary personnel were still allocated to us without us having a say in the matter. Still, we made the best of it by educating and mentoring them. Managements view was still the same, but we tried to communicate testing status in new ways such as using low tech dashboard.

A bit later management had changed in how they worked with the test leads. We were allocated temporary personnel, but we were able to say No to those that we knew were not fit for testing. The cooperation between test lead and testers were very different. Everyone in the team was taking part in making the plan and changing it over time. We found more bugs than before, took pride in our testing, test reports and bug reports. Each artifact delivered needed to be in a good shape so that we could affect the ever so present perception, that testers did not know what they were doing or did not care. In the test team we were all testers, some just had a bit more administration. There was no clear hierarchy.

How does this affect the testing debt?

Having a scripted test approach:

• Does not fit well in an agile and fast moving team or organisation. What are you contributing with to the team? Unclear if you need someone else to write tests for you. Big chance that you have the attitude that you wait for others before you can do your own job. This means that you are a bottle neck and a weight for the organisation. Most of the time you just cost money and is therefore a burden.
• Being viewed upon as just an executor is demeaning. Not having any authority to affect your working situation will eventually mean you give up or stop caring. When you stop caring you will take short cuts and ignore areas. A group of testers who stop caring will affect each other and will make an even worse job. This means it is a catalyst for Broken Windows [4] and tipping point towards the negative.
• When you get new temporary personnel that just weigh you down, it would only seem like you have enough to do the job. In reality you are working even slower and with fewer actual testers.
• When progress is by counting test cases run from one period in time to another, you are missing the point of what is valuable. If the test team is ignorant of this fact no harm might be done, but if they are aware of the fact and dislike the situation, it will cause friction and distrust.

The situations above are extreme, but it is not uncommon, as I see it.

Having a exploratory test approach:

• You are used to have unfinished plans and unchartered terrain in front of you. Living in a chaotic world where you are able to adapt and be flexible to your team and to the organisation. If there is a build available, you test and make the plan as you go. You do not wait. You will rarely be seen as the bottle neck, unless you have too few personnel to do your job. The view of being quick and agile will affect the view on your test team and therefore will make it easier when you start new projects, thus decreasing the testing debt in the area team composition and flexibility.
• Progress is viewed upon as what you spend time on. You then need to justify why you tested that area, but if you do that you gain progress. You know that you can never do all testing, but you might be able to do a lot of what is most important and what is most valuable to the project. By doing it this way, you and the team will gain momentum in your work. You will, if possible, fix Broken Windows or at least not create new ones in this area.
• When you run a test session you know that it is ok to test somewhere else if you think that is valuable. If you find new risks you add them to the list of charters or missions. Your input as a tester is important; you contribute by identifying new risks.
• In a exploratory test team every tester is viewed upon as an intelligent individual, bringing key skills and knowledge. You have no one telling you exactly what to do, but you will have coaches and mentors who you debrief to. There will be a built in training and a natural way of getting feedback. You will be able to identify testers who do not want to test or want to be testers. The team will grow and become better and better. The debriefing will also assist in identifying new risks, keeping the group well aware of current risks and important issues. This will decrease the testing debt by having a focused, hard-working team of testers doing valuable testing, as they themselves see it.

References:

[1] Turning the tide of bad testing – http://thetesteye.com/blog/2010/11/turning-the-tide-of-bad-testing/

[2] A Tutorial in Exploratory Testing – http://www.kaner.com/pdfs/QAIExploring.pdf

[3] Growing test teams: Progress – http://thetesteye.com/blog/2009/10/growing-test-teams-progress/

[4] Broken Windows theory – http://en.wikipedia.org/wiki/Broken_windows_theory