thoughts from the test eye

Project managers have a great impact in our daily work and can often affect how we work, thus what is meaningful or not.

Background

Several years ago I was involved in an organization which was split on several sites in different countries. Project members were not co-located, instead they were split up on different sites. Communication were often a bit strained and confusion because of cultural differences were not uncommon. All test teams reported to project management, who operated from one site and managed testers both at their own site and at other sites.

The project managers wanted to somehow know that testing had good progress. The traditional test process used test cases as a way to report progress. Traditionalists in testing had recommended that the project managers asked questions to the test teams on how many test cases had been executed each day and what the pass/fail ratio was. I believe this is a common, yet dangerous, recommendation.

One of these test teams did not executed as many test cases as the project management wanted. It seemed to them that the test team did not work as much as they should. So, the project management put some pressure on the test team to run more test cases.

The same test team had gotten a new manager of testing. At this time, the new manager contact me about being worried about how his team of testers were conducting their testing. He explained, in order to show progress to the project management, the test team rerun the same test cases several times on the same system with the same version of the system. When the test team reported to project management, they in turn were happy with the new, promising result of a high count of test cases and their pass/fail ratio. The project management was not aware of that the test team were rerunning the same test cases several times. They only looked at the presented metrics and ratio, no details behind it.

Progress

There were many problems with this situation. One of them was “you get what you ask for“. Asking for a high count of test cases and promoting it, will result in a high count of test cases by whatever means possible by test teams. Another problem in this case, is that when we talk about progress of testing they did so in terms of test cases executed. The daily activities in the test team does not consist of only running test cases, instead there are so much more that is being done. Yet another problem is the idea that rerunning the same test cases on the same version of a system would yield a different result.

How do you affect this situation to turn it around? The manager of testing who contacted me, tried to coach the testers in doing things differently by going beyond test cases and the execution of them. I discussed with project management about the current situation, shedding light on what result they got based on the questions they were asking.

A test team have a lot of different activities. Testing or executing test cases is a mere fraction of what we do as testers. Still, our main activity should be to test, in most contexts. To talk about progress and just listen to what testing we do, does not give a full picture. Instead ask progress on test activities and if anything is blocking us, or perhaps if we need help with anything.

Health of the system

A separate question should be about the system, sub-system or whatever that the test team is testing. What is the current health of the system? What issues or bugs do the test team see? What major risks that we did not know about before? What areas are now known, but that we still lack information about?

Information about the system is continuously changing. As new versions of the system are produced, earlier information degrade or might not be valid any more.

Coverage

If progress is asked on test coverage, remember that coverage is linked to models of the system. As testers we employ many, many models and each have their own coverage. We can show progress in the sense that we can show what we know or think we know. We can also show areas that we want to know more about, that we currently know too little about to be able to show anything coverage related. We have questions to the system, the project or a situation that could still be unanswered, we have no idea what it leads to or what is hiding behind it. So talking about coverage in terms of percent becomes a bit absurd in that sense.

Aftermath

After having a long discussion with the project management, it was time to see what changes were to happen. The main project manager directly started to use a different language in his questioning instead focusing on progress, health of the system and coverage. As a bonus, he wanted the testers to explore the system beyond what we currently knew.

The result was astounding, seeing change in motivation, test result and amount of information produced from testing. As I see it, the new way of working followed methods that are proposed by many testers with knowledge in exploratory testing, thus mostly non-traditionalists.