thoughts from the test eye

I stumbled on a Wikipedia-article about Adhocracy today, and it made me think about software development methods and software development organizations.
Here are my thoughts…
Heavy-weight development processes strive for being more and more formal and thereby (intentionally or unintentionally) turning the organization into a bureaucracy.
Whereas light-weight development methods seem to strive for the opposite i.e., striving for a more adhocratic organization.
If this sounds reasonable, it might also seem reasonable that many organizations once have started as an adhocracy (start-up company), then moved more towards being a bureaucracy (expanding, need process and method), to finally find out that they really want to get back to where they came from (creative environment).
Not that I would ever think that you need to go through the bureaucracy to realize that you don’t want to be there; it seems more like the bureaucratic approach happens because the “industry standard” tells you that you need to go there.
So, because I nowadays see myself more as an Adhocrat than a Bureaucrat, I will try to encourage you to not go through the heavy-weight processes in order to realize that all you wanted was to get rid of them.
And I do hope that my children won’t have to experience the bureaucracy before the adhocracy when they one day decide to become software testers like their beloved father.

A project manager, that had no knowledge about testing whatsoever, asked me how he should go about testing? The question was vague since he did not know where to start. I wrote him a quick email listing a few things to consider:

• Create a list of all use cases. For each use case consider possible errors. Identify how these use cases and the errors from these could be tested and verified.
• Create a list of techniques used. Identify common errors for the specific techniques. Identify issues that affect these techniques in one way or another. These issues and common errors can be used as a base for testing.
• Create a list of the major areas of functionality that need to be tested. Create a matrix and categorise the areas in two columns (column one: risk anything goes wrong, column two: cost if anything goes wrong) from 1 to 5, where 1 is most costly or highest risk. Sort the categories.
• Identify important milestones when there are deliverables for testers.
• Identify what deliverables you expect from testing. This can be test reports, bugs. Consider when and how often you want these. Be reminded that all administration will take time from regular testing.
• Which test resources are available and to what percentage. Investigate when they are away and if they have other tasks that they work with.
• What testing has been done so far? What is documented? What did they find?
• Is there a backlog of bugs? Are they well tended and classified? Can they be reused and are they still valid?
• Decide on how you want to work with bugs and enhancements
• Have the testers received all possible documentation needed to start working? This can be support documents, help-files, design/functional specifications and so on. These documents can be used for test ideas.
• What will the test environment look like? Will it be a virtual one or a full customer-like environment with hardware and software?
• Have you considered how many prototypes that should be available for the testers? Consider that many might break. Testers without something to test is a no-no.
• Is there any other configurations that we should consider that is outside the test environment that these builds/prototypes need to run in?

If your testers are new to testing consider this:

• Use the requirements to create user scenarios that in turn can create more test ideas.
• Some tests might need to be planned long before they are executed. Example of this can be EMC testing or Safety testing, that might be performed externally in other test labs.
• System testing should be done by testers, thus not a developer or product manager.
• Do not test everything. Focus on costs and risks first. All areas that go untested should be a calculated risk done by stakeholders or decision makers.
• Test everything. All major functionality should have at least one test. Important areas should have deeper testing while less important can have shallow testing.
• Consider long term investments in testing. Will there be more versions of the product that need even more testing. It might be good to invest in test automation if it gives a ROI. Should test cases and test result be saved for the future, in most cases you would like that.

All of this information is usually structured in the various documents that expert testers are used to such as test plans, test strategies and so on. But for someone that did not know anything about testing I just gave him this.

I got no response from the project manager on this though, so either it was total crap or I scared him away.

How many should be employed?
The quick answer is as many employees that are needed to call the team flexible, but it is not as simple a that. In order to get resources for the team, the team leader usually need to prove he/she needs resources at a certain time. Since the team tries to be as flexible as possible, thus planning short term, it is hard to say when the team need more resources. Naturally planning long term and being less flexible to change is not what the team is about, so short time planning where the team knows for sure what to do over the next two weeks is natural.

The team will know after a time what is a good number to be able to handle all that comes the teams way. The planned test activities might have less priority than the unplannable test activities that comes the teams way.

So in order to be able to be flexible and take on the special tasks while at the same time performing the planned test activities the team need to have enough members to be flexible. When there is less to do, the team will be able to invest in the long term test activities.

Should external resources be used?
If it is possible to have the same external resource available that have extensive knowledge about how the company works and how the projects, products and tasks are performed it is very good investment. But if the testing team need to be strengthened temporarily, it is better to look inside the organisation. Someone being temporary in QA will learn a lot and might contribute to the team in a fruitful way. It is even better to have a communicated plan B that includes certain resources that know full well that they might be used as extra resources for testing. Support personel and developers should be the first in. These extra resources should be educated how to work in QA when the organisation has the least to do, thus making it up to speed when the load is heavy.

Introduction
The flexible testing team is one way of organising the testing team. It will not be the optimal in all situations and it might not fit all organisations or companies. It might not even be possible to fulfill this because of management issues. In any case, this is one way of looking at it and thinking about how to organise the team.

When should the flexible testing team be used?
The following criteria is recommended:
* When there are many unplannable test activities, that might coming more frequently
* When deliverables to the test team does not arrive when they are supposed to do
* When it in general is hard to plan the testing
* When there is no need to promise testing has covered “everything”
* When there is no promise from management that extra resources will arrive if there is an extreme work load

What are the success factors
To ensure that the flexible testing team is successful the following things could be used:
* Test planning is good to do by the week and having planned ahead tasks that are almost certain to happen
* Work off high priority tasks as quickly as possible (using the whole team if possible)
* Remove or postpone tasks that are not really high priority
* Make test preparations as early as possible or when there is slack in the plan
* Do not focus on too much administration and reports
* When speaking with project managers or management be flexible in your attitude
* Try to make project managers or management to be flexible with the tasks for your team

Flexibility
What does the term flexible really mean?
Flexibility according to Wikipedia.
Flexibility to whom or in what way?
It might be that management want the testing team to be flexible about their tasks or it might be that the team itself want to be flexible to be able to handle all tasks. Whatever the cause, it is good to understand who asks the team to be flexible as well as asking why and what it means, thus what it has to achieve to be considered flexible.

Flexibility to undertake anything that comes the teams way?
Sometimes it is impossible to be as flexible as the team would like and on those occations management need to assist with prioritizing tasks. In order to make it clear for the team what to expect and who are dependent on the team, consider the following questions:
* Who are able to use the testing team?
* What has been promised to them?
* Is the team able to say No?
* Is there resources that can be borrowed from other departments if the testing team is under resourced?
* Is there a plan B regarding resources if the team is under resourced still?

Flexibility to work both with short term and long term tasks?
It is important that the testing team is able to catch its breath sometimes. To work on long term test activities that invest for the future is essential for having a functional testing team. When the testing team does not have time for long term test activities during a longer period of time it should be seen as a major risk and that the testing team is under resourced.

The variety of configurations (operating system, browser, language etc.) can look overwhelming; and it is impossible to test all possible configurations for Servers and Clients. On the other hand there are certain platforms that are more probable to uncover defects.

This is just common sense, but I haven’t seen any terminology for handling this in an effecient manner, so let’s call it BCM.
Basic Configuration Matrix is a short list of platform configurations that will spot most of the platform bugs that could exist in your currently supported configuration matrix.

The simplest example is to use one configuration with the oldest supported operating system, oldest browser etc; and one configuration with the newest of all related software. A more advanced example could use several configurations that use different languages, Application Servers, authentication methods et.al.

If it would take too long to run most tests on BCM; alternate between the configurations while testing your product. Do variations on configurations when appropriate.

Actually not, but still it is a must if you have not tried it.

Combine it with exploratory testing techniques and you will be effective and have a good time. One person explores and tests while the other documents, interacts and questions.

Documenter:
* Write down one-liner or at least short blocks of text for each bug found in a document. When time permits report bug in the bug system.
* Categorise the bugs found.
* While documenting you will often come up with new ideas. Just tell the tester directly or make a switch!

Tester:
* Follow the normal principals of exploratory testing.
* Talk with with your documenter each time you find something interesting, keep your focus on testing and exploring.

Now and then switch places so that you can rest your mind from exploring. It is a lot of fun to test together. If you cannot do it at all times, try it at least once. You will learn from each other. There are lots of articles on pair testing, read them and see what else you can do.

Agile development methods are becoming a more popular way of producing software in contrast to traditional project processes. This has affected the testing profession in many ways, which has given us both benefits and new challenges.

In a way, agile development methods can be seen as a reaction to a couple of traditional project risks. But even if some risks have been addressed by these methods, there are certainly new risks that are introduced.
I have more than 5 years of experience working in Agile development teams and I love this way of developing software. Still, I have seen a lot of new risks that has been introduced with these approaches; risks that sometimes are ignored, and sometimes they hit the project with full strike.
So,
* what new risks have been introduced in Agile development?
* And especially, what new risks are introduced for testing in agile projects?
* Which risks are applicable for testers in a project where developers do unit tests and testers are supposed to do acceptance tests?
* Which risks can affect the testing in a project where you end up as the only tester out of 8 team members of a Scrum team?
* Which risks are to be found in an agile project where one test team is supporting several agile development teams?

So, can we (I invite you all to this task) identify and define 5 core risks that are applicable for testing in agile projects?
By core risks, I mean risks that would be so common that they could be considered to be valid in most contexts.

And…

Can we try to find out about how risks can manifest in agile projects and what their transition indicators are?
I guess this would be a good way of learning how to mitigate and take actions to minimize the costs for the risks to materialize.
As risks are something that will always exist in software development, it is better to manage them than to ignore them.

I would like to remind you all about the Sysinternals tool site, which now reside on Microsoft Technet, and is called Windows Sysinternals.
http://technet.microsoft.com/sv-se/sysinternals/default(en-us).aspx

Old favourites like RegMon and FileMon can be found here; so also the new gems PsTools, Process Explorer and Strings.

Great tools for testers on Windows platforms!

I seldom see discussion about our status reports. I refer to the reports that we write for the release, iteration, build and so on. There are several templates and standards out on the internet but I have not read any ground breaking updates in that area for a long time. Many might see the status report as a minor part in what we do and that it should not get too much space in our discussions. It’s the actual testing that is the most important, but I still think it deserves some thought.

As I see it our status reports reflect both facts and feelings. One of the goals is to give the stakeholders as much information as possible to make decisions regarding quality. I have read many status reports that sometimes, as I see it, lie about the situation. Examples of these lies can be that the writer tries to gain respect by exaggerating about certain areas and can sometimes suggest not releasing just because of an internal project vendetta, and so on. Naturally this should never be the case, but I have seen it and I’m sure others have as well.

It is hard to write about status. There are so much that can be included. So, perhaps we could learn more from similar professions that write reports such as these. The professions that I think of first are film critic and food critic. Both of these are preferably experts in writing the status reports, either for a film or a something related to food.

Anyone have further ideas how to perfect our status reports?

I have a friend that proof-reads every bug report before reporting; “otherwise there will be spelling errors”, she says.
I have a friend that always reproduces the bug before reporting it, at least four times.
I have friends that always ask “Bug or Feature?” or “One or two bugs?” when they are uncertain.
I have friends that look beyond each symptom to see if there are more serious errors as well, or if the same type of issue exists elsewhere.

I have a friend that tests the whole area so each bug report can contain all important information, but another friend reports every bug at once, so others get the information fast.
I have a friend that does everything possible to not report Duplicate, As Designed, Not Repro bugs; but another friend thinks that the occasional As Designed issue is a sign of taking responsibility.
I have a friend that focus most on formulating really good titles since they will be read by many, but another one focus on making sure the bug contains as much extra information as possible for the developers.
I have a friend that sometimes makes sure that the report contains only the necessary information; but sometimes puts the issue in a scenario context so the importance is clear.

Bug reports are our most deliverable and our best knowledge base.
And remember what they say in Starship Troopers: “The only good bug is a dead bug”