thoughts from the test eye

With subproject I mean when larger project have a need to split the project into minor parts. The essence of a chicken race is “Each player prefers not to yield to the other, the outcome where neither player yields is the worst possible one for both players.”

The goal for the larger project should be to release in time, with expected quality and so on. The goal for each subproject is usually to do their part and then integrate with the rest. The big problem is when something goes wrong or when one of the subprojects are delayed. In an unhealthy situation the subproject will not tell anyone. They will do all in their power to wait until someone else speaks out that they have been delayed or have problems of some sort. At the project meeting where all subproject managers are asked about delays, quality etc they will hold up a illusion of success. This can go on for a long time. Everyone is waiting for the first one to speak out, letting the rest get some extra breathing room and perhaps some extra time. The whole situation has turned into a chicken race.

The subproject managers will go about each other trying to see if someone is infact late and perhaps bring it to the surface. It is very apparent that everyone might be working towards their subgoals, but very much against the major goal. Naturally this is not always the case, but I have seen it happen many times. It is probably most common when there are many subcontractors and when the company culture promotes this behavior.

My only advice is stay to the truth, as you see it. If you are delayed do tell the others so that they can plan accordingly. In many cases you can change from making the short term decisions which might be costly, to more long term decisions that will be better for everyone. Keeping an openness amonst the subproject managers and über project managers will enable you to move more swiftly. Still, in some cases you might get kicked out as a subcontractor if you are delayed, in those cases you had better have a good plan B.

The front door

The bug that a co-worker found was that you were able to pull open the door a quarter of an inch, just enough to be able to put in a little stick and then wave it infront of the radar. It actually goes quicker to use the stick instead of the card.

I confronted the installer of the security radar and told him about the bug. He answered me “No, that is not possible.”. I told him again, it is so and that one of our consultants use it since he has no pass card. He answered me again “No, that is not possible.”. I did not have time to argue so I left.

I am now going to notify the owner of the house so that they understand the problem. Apparently there are more people using this trick when they have forgotten their card, so the stick is just below the pass card holder for easy access. A few years ago there was something called “Not my job award!”, I guess this would fit into this category.

What is essence of the discussion on release team/showstopper meetings?

I am assuming that there is a meeting. I’ve been to many different kinds of showstopper meetings and most companies handle them differently. One important item on the agenda for the meeting is usually the bugs that are found late in the project, thus at the end phase where they might be considered showstoppers.

When these bugs are discussed if they are to be fixed or not, I’ve seen cases when the argument is a bit vague. The actual topic is sometimes not brought to the surface, instead it is someone who “feels” that this should be fixed or it should not. What is it that they feel? Articulate those feelings and try to be clear on what it is you see. Why should it be fixed or why shouldn’t it?

In these cases I think it is best to be clear that we talk about cost, at least in most cases, in one sense or another, thus the cost of not fixing it or the cost of fixing it. Some costs will affect support, others will affect marketing and sales and some will affect development itself. I can recommend trying to get everyone attending the meeting to focus their thoughts on what the bugs will cost you. Is a few days slip perhaps worth it? In some cases the date for the release is extremely important, this is most often the case for bigger organisations. Instead it is easier to make some changes after the release decision and make a few patches afterwards. The actual release will not reach the customer the same day anyhow. Either way it is cost you discuss and you most often decide based upon it.

Cem Kaner has written an article on this that I’ve used many times when setting up these meetings. You can find the article here.

The first build has been delayed. I was able to test a bit on the last support issue, but it was just a quick one and it was a few weeks ago. I’ve nothing to report bugs on. Should I enter some bugs into the code myself just to quelch my thirst? Should I test some other product, perhaps an open source tool? I crave for testing something. I must have my fix! Give me something to test! I’ve spent a lot of time on planning, collecting test data, configurations and so on, but it is testing that does it for me.

I have become addicted to testing.

It cannot be exactly defined what a bug is, or how it should be reported. And each tester, developer, project manager et.al. has her own way of writing, thinking about and handling bugs.
I like to think of this as taste.

Do you prefer having all details in a bug report, or only including what you think is important?
Would you rather put high priority or low for an in between case?
Can someone else recognize the style your bug reports?
Do you often think your bug reports are more important than others?
Are you often finding the same type of bugs?
Should the first letter in a title be capitalized?
How do you feel when your bug is returned As Designed?
Do you prefer informal reporting for some type of issues or persons?
Do you count your bugs, and if so, why?
Is the audience preferences for bug style more important than your own (efficiency)?
Which bug would you like to find?

Is your taste for bugs impeccable, in your context?

How is information handled around the decisions for the product release?

There are many different situations around this, the following is what I’ve experienced and my tips and tricks.

In many cases there is no information available or rather it is not presented to the decision makers. It also common that the information is not required to form the decision. The decision is formed based on gut feeling or something similar by the decision maker, but not on the information available from the project members.

In some cases information is produced based on IEEE standard test reports. It is the standard report that no decision maker has explicitly asked for it, where they get the standard information based on “best practices”. This report may contain many good things, but also many things that the test report writer should perhaps not spend time on. Consider how much you really want to spend on administration instead of testing. Michael Bolton and James Bach have lots of good stuff in their Rapid Testing material regarding this.

There are also controversies regarding measurements that you want to produce and report. Examples of such statistics could for instance be bugs (bug count, critical bugs, etc), how many tests you have executed, how many scripted tests, how many exploratory tests and so on. The list of possible measurements is almost infinite. When you are using metrics, ask yourself if this really give any valid information that can be used for a decision of some sort. If you use metrics, have it in an appendix and write your own conclusions and context around the metrics.

If you have a standard test report that you “must” use, ask the decision makers which information that you can get rid off and what they want to see there instead. A tailor made report for the decision maker is much more appreciated than a generic or standard report.

If you start from scratch with gathering information for decision making, ask them what information they would like to be able to a make sound decisions. Naturally each decision maker might need different information. All information should not be produced by the testers, which is sometimes the case. It is better that information comes from many sources: unit test coverage from developers, project management details from the project manager, test results and coverage from testers and so on.

There is a controversy around reporting your gut feeling as part of the information for the product release. In a sense I agree that it is good to avoid feelings in your report, only focus on facts. Still, if you write about your gut feeling as one source of information it should be ok. Gut feeling might tell something that you have a hard time explaining, something you cannot point at directly.

Who makes the decision that the product is ready for release?

There are many different cases of this situation, here are a few examples.

In some companies it is the QA department that makes this decision. This means that it is QA that takes the risk for the release. It also means that if something happens after the release it is QA that is to blame for making a “faulty” decision. I’ve been in situation when management said “Why did they not test that?” or “How could they miss that area? It was so obvious.”. If this happens QA will be worried more about what the rest of the company will say and will in some situations say that the product is not ready for release just to avoid possible blame. Does QA have the authority to get longer time to test, are they able to get more bugs fixed, what authority does QA have?

In some cases it is the project itself that makes the decision where the whole group (developers, testers, project manager and so on) decides. This means that the project and most often the project manager him/herself takes the risk for the release. So, the blame will fall on the project and its members. Is the project able to get resources, time etc if need be? In most cases no.

In other cases it is management or a release team consisting of managers from all departments that makes the decision. This means that someone outside of the project makes the decision for the release. It also means the blame will fall on this team, not the project itself. It is also common that this team can affect resources, time etc for the project.

We also have the absurd case where no one makes the decision. The release just happens and no formal decision was made. In this case it is more of a bad process issue that this happens. Management is responsible unless the project manager has the formal responsibility in these situations.

If you are unsure who makes this decision you really need to discuss this with management. Someone must take the risk and responsibility for releasing the product. I recommend that it is management or a release team that makes this decision, since they have the authority to make changes. This naturally leads into the role of QA, are they Quality Assurance or Quality Assistance (as Cem Kaner so nicely put it)? I think that if testers can avoid making this decision it will make their life a lot easier.

When working as a consultant you must live up to the paradox that:
You should already be specialised and competent enough to get a contract i.e. best of all candidates for the job; but in order to stay alive in business you need to be as general as possible in order to meet the criteria for as many contracts as possible.
I.e. you should be a specialist and a generalist at the same time.
No news so far.
And in order to get specialised you can improve a lot by reading, studying, getting hands-on experience, discussing, etc. All made to improve your special skills.
So how about improving your general skills?
You could strive for working in as many different projects as possible; and you could also strive for shifting roles and work assignments.
By getting in touch with several techniques and methods you increase your toolbox, both when it comes to testing techniques and development methods/procedures.
But as I see it, perhaps the hardest thing is to be able to learn as much as possible without filling up your brain with too much noise that you won’t have any use for in future projects or the rest of your life.
So how can you maintain your general skills on a good enough level without being so skilled that you risk becoming a specialist? And especially if this concerns an area in which you only would like to maintain your general skills.
The trick is to be selective when learning stuff.

But notice that there is a difference between Strategic Incompetence vs. Selective Learning; let me give you an example.

A couple of years ago I faced a problem. During the first week on a new project I was told to read hundreds of documents and ten thousands of pages of documentation for a large system in order to understand what it was all about; a system which to me was a new technology and I had never worked with before. So in order to learn more about this system I was supposed to test I had to read these documents.
And my first reaction was “No, I will not do it”; the second reaction was “No, definitely not”. Actually, it is silly to demand that anyone should read this vast amount; and especially if you don’t know how long the contract is going to be. It might have been fine if the contract was on 5 years or so, then it might have been some point and time to complete it. And even if my reaction was “No”, I could not say to the customer that I would refuse to read these documents. Even if it had been right, I didn’t had the guts to do it; not in the first week of the contract.
Now you might think that I was lazy and did not want to take on the challenge to read all documents?
The issue was not that it would take time to read the material; my biggest issue was that if I have to choose what information to store in my brain I would not choose this noisy information. There are so many interesting things to learn; so many interesting books to read; so many lovely live moments to experience. There were so many other things that were much more interesting than the soulless information that I was supposed to read…
So what could I do?
It was not an option to grow my strategic incompetence in the area; then the contract would have been lost.
Instead I chose to read the material as shallowly as possible and only focus on those things that mattered; and thereby try to be as selective as possible when it came to learning. I.e. I developed my selective learning skill…
I consciously refrained from learning in order to not become specialized in the area.

My goal was to be a generalist and still be able to complete my mission. I wanted to preserve my specialist skill in testing but not in the specific subject (area of interest).

Read more:
http://www.cio.com/article/102352/Specialists_vs._Generalists
http://finance.yahoo.com/career-work/article/102876/the-art-of-showing-pure-incompetence-at-an-unwanted-task

Random testing or fuzzy testing is nothing new, but for those of you who are new to it I just wanted to share a little tool I found. If you want to know a bit more about fuzzy testing go read at http://en.wikipedia.org/wiki/Fuzz_testing or whatever place you like to find quick info at.

Barton Miller has written a bit about this and made some binaries to use at:

ftp://ftp.cs.wisc.edu/paradyn/fuzz/

Read ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-nt.pdf for interesting MS Windows results.

The binary subjects an application to a stream of random input either by keyboard and mouse events or by using SendMessage and PostMessage. I used the binary on WinXP on UltraEdit and MS Word. I even tested the feature to simulate mouse click and mouse movement with a few interesting results.

What does this tell me? What would I expect from this? Is this a tool that I want to have in my tool box when testing software?

Using this tool will help you check robustness of an application. It will do automated random testing. If an application crashes often it might be that they have not properly handled Win32 messages. I will try this out in my toolbox. Even if you decide not to use it, read Bartons article.

New RSS: http://thetesteye.com/blog/feed/

The Swedish software testing blog “thoughts from the test eye” changes blog software and will from now on run on WordPress.

The old software didn’t have good enough RSS support, wasn’t appealing to the eye, and had some smaller usability issues that was becoming too annoying.

The start page will still be available at http://blog.thetesteye.com or http://www.thetesteye.com, and all content has been transferred, but direct links to existing blog posts will not have the intended effect.

The blog, which is one of few with several authors, will continue to discuss matters like people, machines, stakeholders, tools, techniques etc.
Read and discuss!

Cheers,
Rikard, Henrik, Martin