This is a follow-up from Working with the testing debt – part 1 [1]. The reason for the clarification is that you so easily come up with a tip without a context or example.

Tip 2: Focus on what adds value to developers, business analysers and other stakeholders. If you do not know what they find valuable, perhaps it is time that you found out! Read Jonathan Kohl’s articles [2] on what he thinks adds value for testing.

In one project I worked with a group of experienced developers. I had not worked with them before close hand, but they had received some of my bug reports and knew me by reputation (whatever that was) in the company. When I got their first delivery to me to test I started right away. Immediately I got the feedback that I was not testing the right stuff and there were a bit chilly in their demeanor towards me. I investigated a bit what had happened and found out that they were not really interested in the minor bugs at that moment and that I should focus on the major issues. I explained to them that I report everything I find, I was not expecting everything I found to be fixed though. What was fixed was up to those who prioritized the bugs. Before I started testing I asked them what they wanted me to focus on first. After that they were a lot happier, both knowing I worked on things valuable to them and that they understood that I reported everything that I found.

During another project we were two weeks from the release of the product. We were in the middle of a transition from traditional scripted testing to an hybrid with both scripted and exploratory testing. Rather, we had test scripts that we used as a guideline when we explored, but we reported Pass/Fail on those. At that time Project Management was strict in wanting number of test cases run as well as the Pass/Fail ratio. Earlier test leads had not communicated well why these figures held no value. When we had run all planned test cases project management communicated to their managers that we were done. But we were not, we continued with working on our planned charters and ran sessions. We interviewed the support organisation, business analysts, product management and experts in the test organisation. Eventually we got a long list of risks and areas that we should investigate. We also got a long list of rumours that we intended to confirm or kill. Basically, we were far from done and we still had time before the release as we saw it. We had also received areas that people in the organisation found valuable to get information about. Still, we failed because we had not communicated enough to project management what we were doing. We managed to go through most of the areas and identified lots of new issues as well as killing many old rumours. We failed to bring value to some, but not all.

How does this affect the testing debt?

If you continue to work on things that have no value to you or any of your stakeholders, you must take a stand and change things. Do not accept the situation as it is. If you and everyone around you think you and your test team are not doing anything of value, it will just add to your testing debt.

As I state above Jonathan Kohl gives a good set of questions [2] for you to ask yourself to get back on the path. Also consider what Cem Kaner writes about in Ongoing revolution if software testing [3], because it is still ongoing and it not over.

References

[1] Working with the testing debt – part 1 - http://thetesteye.com/blog/2011/05/working-with-the-testing-debt-part-1/

[2] How do I create value with my Testing? - http://www.kohl.ca/blog/archives/2010_08.html

[3] Ongoing Revolution of Software Testing - http://www.kaner.com/pdfs/TheOngoingRevolution.pdf

One of Cem Kaner’s many classic writings is “What is a Good Test Case?”
It is a very good article, well-spent time for anyone involved in software testing.
But when writing about test ideas, I started to realize that the list of properties for good test cases isn’t perfect, for me.
So it’s time for some criticism of this part of the professor’s article.

His list of attributes for good test cases (for the context “Tests Intended to Expose Defects”) goes something like this:

• Powerful – a test is more powerful if it is more likely to find bugs
• Yield significant results – the issues found are important (to stakeholders)
• Credible – the actions in the test are realistic (not corner cases “noone woud do”)
• Likely – the test reflects how customers actually will be using the product
• Easier to evaluate – ease of telling if the test passed or failed
• Useful for troubleshooting – so it is easy to find out what went wrong during the test
• Informative – regardless of pass/fail status, you get valuable information from the test
• Appropriately complex – if there are many bugs in the software, the test might fail too quickly
• Giving insightful information – the test might not render bugs, but other important information

Test cases can be seen as a broad spectra, from “classic” test cases with exact steps and expected results, to vague, one-liner test ideas (that also could be ongoing, unorthodox or unverifiable.)
So I have to disagree with “Easier to evaluate”, it’s a valuable property in many situations, but not in so many that it should make it on this list. But it’s back on the list if it is named something like “Hints for Evaluation”.

But now to the most interesting part of reviewing: what’s missing on the list?
There are many more things that could be important, a quick search gave:

• Ease of maintenance, Ease of creating variations of it (Shrini Kulkarni)
• Accurate, Economical, Reusable, Tracable, Self-cleaning (Dianne L. Runnels)

These are good things, but not needed on my list (and I’d prefer Fast to execute as name instead of Economical.)
But I have two other things I would like to add

• Easy to understand – to enable reviewing by different types of people (more likely to happen for one-liners)
• Enables serendipity – the test is rich in the sense it has possibilities of finding issues other than the ones the test case is aiming for

The serendipity can either be covered inside the test case, or by allowing variations, or by putting freedom and responsibility on all testers to deviate and look at more things, if deemed worthwile, while executing the test cases.

…and always remember Kaner’s advice that “Test cases can be “good” in a variety of ways. No test case will be good in all of them.”

Well, to many people this is nothing new. But still, there are a lot of testers, and indeed test leads, that still think that Exploratory Testing is a technique that can be used in testing. To some extent, it has to do with that both Cem Kaner and James Bach have used this term amongst other techniques (e.g., in the BBST course material). But they have changed and updated presentations as much as possible over the last period of time.

According to Cem Kaner nowadays, the definition of exploratory testing is “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

And this is important. You can come a long way on reaching the style of Exploratory Testing just by treating testers as intelligent people; which is one of the most important factors in the definition above. In contrast to Exploratory Testing you have Scripted Testing that, in my opinion, treats testers as dumb people or even dumb machines. I think that this approach is devastating for our profession (even though I can somehow see the need for Scripted Testing in some places).

A technique is a recipe for solving a problem, whereas a style (or approach) is a way of thinking around a theme that stretches far beyond solving a particular problem.
So when we talk about selling in Exploratory Testing to managers or project stakeholders it is not a technique that we are selling; it is rather an acceptance of a mindset where testers are treated as professional and intellectual human beings that are able to perform Sapient Testing, and particularly in an Exploratory way. It is not about stakeholders investing in a technique, it is about them showing that they have as much trust in testers as they have in other intelligent co-workers of the project.

As a user of Facebook I feel really helpless when nothing works as it should (as was the case with the latest GUI-update). Posts were stochastically shown in the feed and a lot of errors occurred in various situations. A lot (all?) of my friends on Facebook experienced the same problems.

When there are lots of bugs on flight booking sites I get so frustrated and angry because I cannot complete my task. E.g., the booking system for Ryanair (at least two years ago when I last used their booking system and never will use again).

But what can I do as a frustrated enduser?

The problem nowadays is that you as a user and consumer don’t know what to do with your complaints. The companies shows no respect as long as the users are hooked on the application and needs the solution to problem that the application provides. But they do not care to fix the bugs that might ranging from annoying to critical.
So the loser is you! And it becomes a rage against the machine because of the absence of real people to talk to. Or some people might think that they did something wrong when it really was lousy software that caused the problems.

But in fact all other users of these applications experience similar problems. So you as an enduser is not alone! This is something that hits many people on their own but who lacks a community i.e., if we were employees in a company using the software we would have protested.

What should we do!?

Should we create a site similar to http://www.badsoftware.com/ (Kaner & Pels) or http://www.badsoftware.com/alienwaresucks/ (Kaner) in order to channel some of the frustration and gather it into some powerful criticism?

Or are there other channels that we could utilize?

Do you have any ideas?

Cem Kaner has updated his set of publications. I’ve been reading his well written articles over the last ten years.

Have a nice time digging in!

I use Jerry Weinberg’s definition of quality: “Quality is value to some person”; and I use Cem Kaner’s extension to the definition so that it becomes “Quality is value to some person (that matters)”…

I.e. quality is inherently subjective. And there are a lot of persons that are affected by software that we produce… With this in mind it becomes hard for a tester to stay focused when there are so many persons with opinions that could matter; but if we can find out “who matters” we decrease the number of possible values to care about. Still, this will leave us with several important values that need to be taken into account when testing the product.

So how can we testers deal with that?

You could do a role play when testing and put on someone’s hat during the test session; or you could let real users test the product and let them have a say about what they find.
But for a skilled tester it is more about being multidimensionally subjective and think as several persons at the same time.

This means that a lot of values, beliefs and preferences are taken into account which might matter. Not as an average, but as several independent quality dimensions that has (more or less) importance. The hard thing is to know when a value is threatened and for which (type of) person that is affected; and if this matters at all.
I.e., it is a matter of questioning “is there a problem here?” constantly and try to pair a potentially threatened value with its corresponding person. And if this problem threatens a value for some person that matters, we have found a bug. This corresponds to the definition of bug from Cem Kaner “A bug is something that threatens the value of a product”.

Much of this happens automatically for many of you skilled testers out there; when I thought of it recently I realized that this is something I do more and more and hopefully I am improving this skill each day. This is a great skill to have when testing software!

Anyone having any thoughts on this?
Have you experienced this yourself?
If not, does it sound like an interesting thing to examine? Would this be helpful to you?

Cheers,
Henrik

Update 2009-09-14: According to comment from Michael Bolton, see below, the quotes that I said belonged to Cem Kaner are both quotes from James Bach. I apologize for referencing wrong person.

Well, “secretly” as in that he does not know that I am in love with him… Yet!

If you haven’t discovered the amazing Cem Kaner yet, I can give you the following advices and hoping that you too might fall in love some day:

• Visit kaner.com publications and read ANY article from his large publication-section.
• Buy and read any of his fine books (I would especially recommend Lessons Learned in Software Testing)
• Follow his blog on http://www.satisfice.com/kaner/ (though he updates too rarely)!
• If you haven’t got the possibility to see him lecture in real life, take the BBST online course! http://www.testingeducation.org/BBST/
  And yes, it is for free!
• Read his words about Context-Driven Testing and what it mean; and what it doesn’t mean: http://www.context-driven-testing.com/
• Join the context-driven testing forum and read his stunningly thoughtful comments: http://groups.yahoo.com/group/software-testing/

Are you still not convinced about his greatness, I urge you to talk to any test expert in our industry and see what they have to say about him.
Note: A lot of work above is collaborations with other great people in our industry e.g., James Bach, but they don’t really touch me as much as Cem do.

What is essence of the discussion on release team/showstopper meetings?

I am assuming that there is a meeting. I’ve been to many different kinds of showstopper meetings and most companies handle them differently. One important item on the agenda for the meeting is usually the bugs that are found late in the project, thus at the end phase where they might be considered showstoppers.

When these bugs are discussed if they are to be fixed or not, I’ve seen cases when the argument is a bit vague. The actual topic is sometimes not brought to the surface, instead it is someone who “feels” that this should be fixed or it should not. What is it that they feel? Articulate those feelings and try to be clear on what it is you see. Why should it be fixed or why shouldn’t it?

In these cases I think it is best to be clear that we talk about cost, at least in most cases, in one sense or another, thus the cost of not fixing it or the cost of fixing it. Some costs will affect support, others will affect marketing and sales and some will affect development itself. I can recommend trying to get everyone attending the meeting to focus their thoughts on what the bugs will cost you. Is a few days slip perhaps worth it? In some cases the date for the release is extremely important, this is most often the case for bigger organisations. Instead it is easier to make some changes after the release decision and make a few patches afterwards. The actual release will not reach the customer the same day anyhow. Either way it is cost you discuss and you most often decide based upon it.

Cem Kaner has written an article on this that I’ve used many times when setting up these meetings. You can find the article here.