thoughts from the test eye

At EuroSTAR 2008 in Haag I will present “Testing is an Island – A Software Testing Dystopia”. The paper can be downloaded at http://www.thetesteye.com/papers/redgren_testingisanisland.doc

The inspiration was the theme of the conference: “the future of software testing”; and I couldn’t stop seeing a very boring profession, where numbers and so-called objectivity is more important than people and feelings.
There are trends and silver bullets that miss our secret: we perform simultaneous verification and validation.

Test patterns, quality patterns, Q-patterns or whatever we wish to call it. I am referring to test ideas that can be reused in similar contexts. This could for instance be File Handling, Data Types, Installation, Upgrades etc. There is nearly an infinite list of areas that could test patterns could be available for.

When you start a new project with test areas that you are not all familiar with… you wish to have some test ideas or test patterns that you might be able to apply easily. Experienced testers usually have this documented or in their heads. The test patterns created for one company is usually proprietary, so you are not allowed to bring it with you.

So when you start from scratch at a company there is a lot to wish for. You try to use what you have and need to adopt to possibly new systems. Eventually you will notice that you have almost started from scratch. You start redoing the same things that you have done before at previous companies.

I have asked some of the major test consultancy companies about how they do with test patterns and test data. So far all of them has looked at my strangely and said that it is not possible to reuse the test cases they produce. Also, that these test cases are property of the customer.

Imagine that consultant companies with a few thousand testers start assisting each other with test data, test patterns and experiences. Generally, it seems like they are not doing this at the moment. The smaller companies would have a hard time to compete if this was the case.

If there was a public website where these kinds of data could be stored it would make things easier for all testers. Perhaps it is a dystopia, but I see no reason why we need to rethink our test ideas each time we get to a new company. I can understand that these test patterns in themselves hold great value to the company and that they would like it to be theirs alone. Still, I think it should be possible to have the general areas more public.

If anyone have thought of this before and know of a place where to find these so called public test patterns and test data please assist me in pointing me in the right direction. If not, is this something we should take the initiative in starting?

I need to recommend a great book by Rajiv Chandrasekaran: Imperial Life in the Emerald City.
The reason I recommend this book on this blog is that the more I think of it, the more I think it resembles the way how certain processes or methods are implemented in the sofware industry without taking the context into account.
I.e. this book is a great example on how to implement something in a non-context-driven fashion!

Right now, somewhere in the world, test processes are implemented:
* that are not tailored to the needs of the majority of the affected people;
* that will make it harder for the people involved to do a good job;
* that look ridiculous;
* that are best practices according to XXX;
* because of political reasons;
* because someone said so;
* etc…

When reading this book, you will sometimes laugh out loud.
But at the same time you wanna cry when you realize that this actually have affected thousands of people and indeed a whole nation. It is really scary…

Below is an excerpt from the authors’ site http://www.rajivc.com/book.htm

“Imperial Life in the Emerald City is an unprecedented account of life in Baghdad’s Green Zone, a walled-off enclave of towering plants, posh villas, and sparkling swimming pools that was the headquarters for the American occupation of Iraq.The Washington Post’s former Baghdad bureau chief Rajiv Chandrasekaran takes us with him into the Zone: into a bubble, cut off from wartime realities, where the task of reconstructing a devastated nation competed with the distractions of a Little America—a half-dozen bars stocked with cold beer, a disco where women showed up in hot pants, a movie theater that screened shoot-’em-up films, an all-you-could-eat buffet piled high with pork, a shopping mall that sold pornographic movies, a parking lot filled with shiny new SUVs, and a snappy dry-cleaning service—much of it run by Halliburton. Most Iraqis were barred from entering the Emerald City for fear they would blow it up.

Drawing on hundreds of interviews and internal documents, Chandrasekaran tells the story of the people and ideas that inhabited the Green Zone during the occupation, from the imperial viceroy L. Paul Bremer III to the fleet of twentysomethings hired to implement the idea that Americans could build a Jeffersonian democracy in an embattled Middle Eastern country.”

At www.context-driven-testing.com you can read “Metrics that are not valid are dangerous.”
I believe this is true, but I would rather prefer “Metrics are dangerous.”

Uninterpreted measurements are not bad by themselves, but when value is added to them, they become metrics, and dangerous because they state specific things without considering a lot of other things, that actually might be much more important.
If measurements are used together with knowledge of details, you might have an analysis that is fruitful.
But at many times, sound judgment not only is enough; it is better.

Measurements/Metrics are probably useful for dead things, like manufacturing objects, but m/m are not good for complex things involving people, e.g. software testing. Metrics uses numbers to reduce the complexity and thereby the “truth” disappears.

I stumbled on a Wikipedia-article about Adhocracy today, and it made me think about software development methods and software development organizations.
Here are my thoughts…
Heavy-weight development processes strive for being more and more formal and thereby (intentionally or unintentionally) turning the organization into a bureaucracy.
Whereas light-weight development methods seem to strive for the opposite i.e., striving for a more adhocratic organization.
If this sounds reasonable, it might also seem reasonable that many organizations once have started as an adhocracy (start-up company), then moved more towards being a bureaucracy (expanding, need process and method), to finally find out that they really want to get back to where they came from (creative environment).
Not that I would ever think that you need to go through the bureaucracy to realize that you don’t want to be there; it seems more like the bureaucratic approach happens because the “industry standard” tells you that you need to go there.
So, because I nowadays see myself more as an Adhocrat than a Bureaucrat, I will try to encourage you to not go through the heavy-weight processes in order to realize that all you wanted was to get rid of them.
And I do hope that my children won’t have to experience the bureaucracy before the adhocracy when they one day decide to become software testers like their beloved father.

A project manager, that had no knowledge about testing whatsoever, asked me how he should go about testing? The question was vague since he did not know where to start. I wrote him a quick email listing a few things to consider:

• Create a list of all use cases. For each use case consider possible errors. Identify how these use cases and the errors from these could be tested and verified.
• Create a list of techniques used. Identify common errors for the specific techniques. Identify issues that affect these techniques in one way or another. These issues and common errors can be used as a base for testing.
• Create a list of the major areas of functionality that need to be tested. Create a matrix and categorise the areas in two columns (column one: risk anything goes wrong, column two: cost if anything goes wrong) from 1 to 5, where 1 is most costly or highest risk. Sort the categories.
• Identify important milestones when there are deliverables for testers.
• Identify what deliverables you expect from testing. This can be test reports, bugs. Consider when and how often you want these. Be reminded that all administration will take time from regular testing.
• Which test resources are available and to what percentage. Investigate when they are away and if they have other tasks that they work with.
• What testing has been done so far? What is documented? What did they find?
• Is there a backlog of bugs? Are they well tended and classified? Can they be reused and are they still valid?
• Decide on how you want to work with bugs and enhancements
• Have the testers received all possible documentation needed to start working? This can be support documents, help-files, design/functional specifications and so on. These documents can be used for test ideas.
• What will the test environment look like? Will it be a virtual one or a full customer-like environment with hardware and software?
• Have you considered how many prototypes that should be available for the testers? Consider that many might break. Testers without something to test is a no-no.
• Is there any other configurations that we should consider that is outside the test environment that these builds/prototypes need to run in?

If your testers are new to testing consider this:

• Use the requirements to create user scenarios that in turn can create more test ideas.
• Some tests might need to be planned long before they are executed. Example of this can be EMC testing or Safety testing, that might be performed externally in other test labs.
• System testing should be done by testers, thus not a developer or product manager.
• Do not test everything. Focus on costs and risks first. All areas that go untested should be a calculated risk done by stakeholders or decision makers.
• Test everything. All major functionality should have at least one test. Important areas should have deeper testing while less important can have shallow testing.
• Consider long term investments in testing. Will there be more versions of the product that need even more testing. It might be good to invest in test automation if it gives a ROI. Should test cases and test result be saved for the future, in most cases you would like that.

All of this information is usually structured in the various documents that expert testers are used to such as test plans, test strategies and so on. But for someone that did not know anything about testing I just gave him this.

I got no response from the project manager on this though, so either it was total crap or I scared him away.

How many should be employed?
The quick answer is as many employees that are needed to call the team flexible, but it is not as simple a that. In order to get resources for the team, the team leader usually need to prove he/she needs resources at a certain time. Since the team tries to be as flexible as possible, thus planning short term, it is hard to say when the team need more resources. Naturally planning long term and being less flexible to change is not what the team is about, so short time planning where the team knows for sure what to do over the next two weeks is natural.

The team will know after a time what is a good number to be able to handle all that comes the teams way. The planned test activities might have less priority than the unplannable test activities that comes the teams way.

So in order to be able to be flexible and take on the special tasks while at the same time performing the planned test activities the team need to have enough members to be flexible. When there is less to do, the team will be able to invest in the long term test activities.

Should external resources be used?
If it is possible to have the same external resource available that have extensive knowledge about how the company works and how the projects, products and tasks are performed it is very good investment. But if the testing team need to be strengthened temporarily, it is better to look inside the organisation. Someone being temporary in QA will learn a lot and might contribute to the team in a fruitful way. It is even better to have a communicated plan B that includes certain resources that know full well that they might be used as extra resources for testing. Support personel and developers should be the first in. These extra resources should be educated how to work in QA when the organisation has the least to do, thus making it up to speed when the load is heavy.

Introduction
The flexible testing team is one way of organising the testing team. It will not be the optimal in all situations and it might not fit all organisations or companies. It might not even be possible to fulfill this because of management issues. In any case, this is one way of looking at it and thinking about how to organise the team.

When should the flexible testing team be used?
The following criteria is recommended:
* When there are many unplannable test activities, that might coming more frequently
* When deliverables to the test team does not arrive when they are supposed to do
* When it in general is hard to plan the testing
* When there is no need to promise testing has covered “everything”
* When there is no promise from management that extra resources will arrive if there is an extreme work load

What are the success factors
To ensure that the flexible testing team is successful the following things could be used:
* Test planning is good to do by the week and having planned ahead tasks that are almost certain to happen
* Work off high priority tasks as quickly as possible (using the whole team if possible)
* Remove or postpone tasks that are not really high priority
* Make test preparations as early as possible or when there is slack in the plan
* Do not focus on too much administration and reports
* When speaking with project managers or management be flexible in your attitude
* Try to make project managers or management to be flexible with the tasks for your team

Flexibility
What does the term flexible really mean?
Flexibility according to Wikipedia.
Flexibility to whom or in what way?
It might be that management want the testing team to be flexible about their tasks or it might be that the team itself want to be flexible to be able to handle all tasks. Whatever the cause, it is good to understand who asks the team to be flexible as well as asking why and what it means, thus what it has to achieve to be considered flexible.

Flexibility to undertake anything that comes the teams way?
Sometimes it is impossible to be as flexible as the team would like and on those occations management need to assist with prioritizing tasks. In order to make it clear for the team what to expect and who are dependent on the team, consider the following questions:
* Who are able to use the testing team?
* What has been promised to them?
* Is the team able to say No?
* Is there resources that can be borrowed from other departments if the testing team is under resourced?
* Is there a plan B regarding resources if the team is under resourced still?

Flexibility to work both with short term and long term tasks?
It is important that the testing team is able to catch its breath sometimes. To work on long term test activities that invest for the future is essential for having a functional testing team. When the testing team does not have time for long term test activities during a longer period of time it should be seen as a major risk and that the testing team is under resourced.

The variety of configurations (operating system, browser, language etc.) can look overwhelming; and it is impossible to test all possible configurations for Servers and Clients. On the other hand there are certain platforms that are more probable to uncover defects.

This is just common sense, but I haven’t seen any terminology for handling this in an effecient manner, so let’s call it BCM.
Basic Configuration Matrix is a short list of platform configurations that will spot most of the platform bugs that could exist in your currently supported configuration matrix.

The simplest example is to use one configuration with the oldest supported operating system, oldest browser etc; and one configuration with the newest of all related software. A more advanced example could use several configurations that use different languages, Application Servers, authentication methods et.al.

If it would take too long to run most tests on BCM; alternate between the configurations while testing your product. Do variations on configurations when appropriate.

Actually not, but still it is a must if you have not tried it.

Combine it with exploratory testing techniques and you will be effective and have a good time. One person explores and tests while the other documents, interacts and questions.

Documenter:
* Write down one-liner or at least short blocks of text for each bug found in a document. When time permits report bug in the bug system.
* Categorise the bugs found.
* While documenting you will often come up with new ideas. Just tell the tester directly or make a switch!

Tester:
* Follow the normal principals of exploratory testing.
* Talk with with your documenter each time you find something interesting, keep your focus on testing and exploring.

Now and then switch places so that you can rest your mind from exploring. It is a lot of fun to test together. If you cannot do it at all times, try it at least once. You will learn from each other. There are lots of articles on pair testing, read them and see what else you can do.