thoughts from the test eye

This was originally meant as an answer to the (ironic) thread http://thetesteye.com/blog/2009/06/long-live-the-waterfall/ where a new thread was forked when Ola Janson launched a couple of questions regarding agile development. My answers and thoughts on those questions are listed here.

In one reply to Ola, Rikard says that he has “…never worked in a truly Agile project…” but what is a “truly Agile” project really?

I have worked in both agile (quick and well-coordinated in movement) and Agile (http://www.agilemanifesto.org/) projects.
Here are my thoughts and observations.

• Being agile as a team could apply to any jelled team with a developed group dynamic which manages to quickly respond to problems that might arise. It could also be applicable for any team that has ability to minimise the cost of change instead of trying to avoid the change. I have been in such teams.
• Being Agile as a team is when the team agrees upon the Agile Manifesto (Note: remember that the Agile Manifesto is a set of values and not a set of laws). I have been in such teams.
• XP, Scrum, DSDM, Lean, Kanban, etc are process tools in that they help you work more effectively by, to a certain extent, telling you what to do. These process tools must live up to the Agile Manifesto and are described practises for how to work in an Agile fashion more or less structured; each method has its benefits and drawbacks and none of them are comprehensive. That is why you often see combinations of different “Agile” processes e.g., XP (developing method) and Scrum (team management method) and Continuous Integration (incremental build method).
• You can work in an agile team and still utilize Agile methods such as Scrum or XP; but you could also use other non-Agile methods and still be agile.
• You can work in an Agile team and don’t utilize any of the big recognized “Agile” methods.
• Being a tester in an agile team (or project) is gold.
• Being a tester in an Agile team (or project) can be a smooth experience but it could also be painful. My experience is that if you as a tester are not engaged or involved in the team you will have difficulties. E.g., since documentation is not prioritised you need to know more about the software continuously and therefore there will be problems if you are not engaged in the day-to-day development of the project. And many times the team has been forced into working according to a specific Agile method that does not seem to be suitable for the context of project; and it might be very well suited for programmers but not at all suitable for the testing tasks.

The key thing with the agile movement, as I see it, is that the methods on how to develop software have evolved from the project context and, perhaps most importantly, being defined by the team members themselves. Therefore I believe that those projects have had a successful outcome, at least from the team-perspective. Many times it has correlated with business success.

The Agile movement on the other hand, which originally has sprung from the agile movement, has become more and more strict and promoting “best practises” on how to work Agile. This has become the new cash cow for many consultant firms and they of course promote and teach the “real way” of doing things. But I guess that many founders of the Agile Manifesto really think that it should be more agile than Agile…
So nowadays there are, and will be more and more, voices raised which question the new face of agile/Agile and instead promote the “original” thoughts; or at least the thoughts that they believed in then and still do.

My conclusion is: Find out what matters for you; your team; and project. Define a process that will work for you.

Read more about Agile vs. agile:

• Agile Versus agile Development – http://www.agilejournal.com/articles/columns/from-the-editor-mainmenu-45/18-agile-versus-agile-development
• Defining Agile Methodology – http://www.satisfice.com/blog/archives/45

Find out if your team is agile (or is it Agile?)

• Back-of-a-Napkin Agile Assessment – http://testobsessed.com/2008/11/28/back-of-a-napkin-agile-assessment/

Examples of reactions on current Agile methods:

• Artisanal Retro-Futurism crossed with Team-Scale Anarcho-Syndicalism – http://arxta.net/
• Cargo Cult Agile – http://www.exotribe.com/node/16
• WAgile – http://www.parlezuml.com/blog/?postid=708
• Who Stole Agile? – http://www.satisfice.com/blog/archives/51

Group dynamics

• Read about “Jelled team” (and other interesting topics) in Peopleware – http://www.dorsethouse.com/books/pw.html

From a managerial perspective without knowing too much about testing, your sole experience comes from the scripted test environment…

What does Scripted Testing include?

• Control over what is to be tested, in the sense that you have a clear coverage of test cases on certain areas. Reports where you can see exactly how many test cases are left to do before you are finished.
• Hierarchy among testers, with different roles playing their part in the system. An excellent way to have career moves. Test leads who do the thinking and the others to execute.
• Scalability in personel by easily bringing in new people who can execute the test scripts that the test leads have written. Getting new resources in the middle of a project is easy since you can take almost anyone.
• Test Management Software. If you want your testers to have the best tools, there are tools that cost several millions. They will generate excellent reports, they can automate so that the boring repeatable tests do not have to be rerun manually.
• Education. There are lots of certification to be had and many firms who can teach you the essentials.

What does Exploratory Testing (ET) include?

• Flatness in organisation. The testers make tests as they go along. They do not need the traditional test leads. It is uncertain what kind of career exits you have with this kind of focus.
• Chaos. You have no idea on how you are going to test. You have not planned everything out in detail before you start testing. You cannot report exactly how long time you need and as much time as possible is not good enough.
• No scaleability. You only want testers. Not anyone can be a testers. It is hard to just get anyone to help out since you cannot use just anyone from the organisation, they must know the basic skills to help out.
• No real education. You cannot get certified. None of the major consultants teach this. There are courses, but they are new and few teach it.
• No real Test Management Software. The major software vendors have no tools for this. The software is seldome made by commercial vendors.

What do your managers think about this? What sales pitch do ET have for management? I’ve twisted what ET can include to bring out the worst of it, but I think that an inexperience test manager with his roots in the scripted test environment has at least part of this view. Cem Kaner has a good presentation called A Tutorial in Exploratory Testing. That presentation do sell ET really well. If you are going to try to get your organisation to start using ET and intend to do some propaganda for your test manager Kaners presentation is a good start.

You are on a quite large company where there are several QA divisions, several layers of management, several listeners to each step of the development process. It is the final weeks of the release. You are about to enter a bug which seem serious but you are not sure. You can take at least two paths here…

Take the easy path and you report what you have seen so far and scribble down a few quick repro steps on what you think might be the bug. You do not dig deeper cause its been a long week and it is your right to go home now.

… or …

Take the harder path and you really try to hone down what is wrong. You try to find some repro steps on what it is. You try to see how often this bug appear and what impact you think it has. You extract log files, config files and all other information that you think might help the developer. You proof read the bug report and make sure you have gotten everything in there. You let someone else take a look before entering it.

As the submit of the bug takes place you see that there were many who got the bug in the mailboxes as a final delivery at the end of the week.

Chosing the easier path might have come to a situation where you as an observer…

• do not believe the bug
• ignore the bug because it seem too vague
• think the testers are crap and become unsure if their previous bugs were infact not correct as well
• are scared that the quality of the release is in danger because you have no trust in the bugs
• become angry because the testers do not take their job seriously and enter a bug in such bad state in such a critical moment
• become angry as a tester thinking that the other test team gives you a bad reputation

The list could go on for some time…

Chosing the harder path might have come to a situation where you as an observer…

• are thankful as a developer that you did not have to work the weekend to try to find the reprosteps
• are thankful as a project manager that you were able to make a bug classification more easily before everyone had left for the weekend
• are impressed as a manager that the testers give the extra effort to save so much time for others
• are thankful as a tester that the other testers take pride in their work and deliver at every step

The easy path is quicker while the harder path takes a bit longer time. Depending on your thouroughness and completeness the result will differ enourmously. Naturally this is not always the case, but I’ve seen this happen so many times.

When you come up with a new test idea, you are using your knowledge and experience, but there is also some sort of stimuli that triggers the idea. Something you see, hear, understand or think about.

You seldom think in totally new ways, you rather combine things in a new way.

These are my favorite test idea triggers:

* think about each feature (and connect with testing or technical knowledge)

* create or understand a model of the software

* think about interaction between items in the system

* thinking about quality attributes (CRUSSPIC STMPL + Accessibility)

* read bug report titles for similar functionality

* read test idea lists

* play around with the software, or a prototype, or a previous version

* change perspective, see your software as a little box among many others

What triggers you?

Do you get help from Session Tester Primers?
(http://sessiontester.openqa.org/download.html)

A cheer to those of you who were able to attend this conferance: http://www.waterfall2006.com/

My favorites:

http://www.waterfall2006.com/crispin.html

http://www.waterfall2006.com/jeffries.html

Thank god everyone is not a believer of the hype around the Agile Movement. Process is king!

Well, “secretly” as in that he does not know that I am in love with him… Yet!

If you haven’t discovered the amazing Cem Kaner yet, I can give you the following advices and hoping that you too might fall in love some day:

• Visit kaner.com publications and read ANY article from his large publication-section.
• Buy and read any of his fine books (I would especially recommend Lessons Learned in Software Testing)
• Follow his blog on http://www.satisfice.com/kaner/ (though he updates too rarely)!
• If you haven’t got the possibility to see him lecture in real life, take the BBST online course! http://www.testingeducation.org/BBST/
  And yes, it is for free!
• Read his words about Context-Driven Testing and what it mean; and what it doesn’t mean: http://www.context-driven-testing.com/
• Join the context-driven testing forum and read his stunningly thoughtful comments: http://groups.yahoo.com/group/software-testing/

Are you still not convinced about his greatness, I urge you to talk to any test expert in our industry and see what they have to say about him.
Note: A lot of work above is collaborations with other great people in our industry e.g., James Bach, but they don’t really touch me as much as Cem do.

This post was triggered by blog post Resolved as Not Repro – http://thetesteye.com/blog/2009/06/resolved-as-not-repro/
I believe that bug systems too often are used with onlý a this-project-right-now approach, where you care most about just getting all items dealt with.
This is perfectly fine for one-off type of projects, but does not work fully for software where the bug system is (one of) the most important sources regarding quality information.

Since I use a holistic-subjective approach to testing, I need to consider both now-and-then, and also me-and-everyone-else.
Testers, developers et.al will treat bugs differently if they are resolved As Designed, or Fixed, or External, or Worksforme, or Not Repro, or Invalid, or Postponed, or Won’t Fix or Duplicate. 
At the end of this project, or next version, I might look again at all bugs set as Not Repro, or re-verify the most important Fixed bugs.
If there are many By Design issues, some usability folks might look into the details, and consider new approaches for the area.
Support people might search for information, and act differently depending on the solution of the related bug.
And also people that think that metrics matter, might be even more deceived, if the resolutions aren’t the correct ones.

My point is, you can’t really know how the information may be used, so do your best.

Lets say that you have a bug system; and for each bug you have the two fields “State” and “Resolution” where the following values are valid:
State: New, Assigned, Resolved, Closed.
Resolution: Fixed, Invalid, Won’t fix, Duplicate, Not Repro.

Further, you have a field where a product version number should be entered; i.e., the earliest version number of the product where the bug can be reproduced in.

Now, lets say that you report a serious crash bug that is reproduced in the latest build of the product e.g., version 1.1.2. (Oh, I forgot to mention that you work as a software tester at a software company and this happens during development of the upcoming release of your product.)
Two weeks later the bug is resolved by the assigned developer; the State and Resolution is set to “Resolved – Not Repro” and the following comment is added by the developer in the bug:
“I cannot reproduce this anymore (code is same as in build 1.1.4). This might have been fixed when other bugs in the same area were fixed. Resolving as Not Repro…”

So what do you think?

Do you think that this bug should have been resolved as Not Repro?
If not, what resolution would you have chosen?
Would it make any difference if the resolution “Works for me” had been a valid resolution instead of “Not Repro” (as in Bugzilla)?
Any other thoughts on this?

At EuroSTAR 2009 I will present “More and Better Test Ideas“; the main idea being that testers could generate many different types of test ideas, and communicate them in a condensed one-liner format.
If you have great tips on how to come up with really good test ideas, or want to review the paper I’m about to write, let me know.

/Rikard

Recently in Sweden there was a tragic death to a young child that could have been rescued if only the child had come to a hospital in time for a full exam. The one that was blamed for this death was the medical care hotline company that did not understand the severity of the illness and did not send this kid to the hospital. (Read more, in Swedish: http://www.dn.se/sthlm/pojke-dog-efter-rad-att-inte-aka-till-sjukhuset-1.859096)

After this tragic accident, it was discovered that this private medical care hotline company pays out a monthly bonus to those nurses that keep their phone calls short. (Read more, in Swedish: http://www.dn.se/sthlm/skoterskor-far-bonus-for-snabba-rad-1.864534 )
I.e. if they keep the call below 3.48 minutes and during that time complete the medical record, they receive a bonus of 1000 Swedish kronor (approx. € 100). In order to receive the bonus, there are some quality goals as well. E.g., you don’t get the bonus if you unnecessarily send someone to the emergency ward; or if you give a faulty medical advice.
Do I need to tell you that the county council paid the private company by the number of calls they handled.

This is what happens when you use simplified and dangerous metrics as a foundation for incentive pay… And these metrics are easy to abuse because they are based on simplified models of how the real world looks like.
When dealing with people, you are dealing with “complex systems” (read more in An Introduction to General Systems Thinking, by Gerald M. Weinberg ) and you cannot treat every person like they would be the same. I.e., the people calling in (and indeed children that cannot speak for themselves) are treated as a neutral “* 1” or “+ 0” in the metrics equation.
This happens if you include simplified metrics to measure your efficiency when dealing with people; metrics that leaves out the most important and complex parts of the equation: humans and human interaction.
Nurses know how to work with people, they know that people are unique; they know that their job is hard and requires skill and years of experience. They know that some patients require 20 minutes before they are calm or they need such time to explain everything important; they also know that some people just need 25 seconds before they are satisfied.
It is a shame that nurses are measured by how fast they finish a phone call.

It is the same thing that happens again and again in software industry; or rather the peopleware industry. People that work with developing software are measured by metrics that are dangerous and wrong; and in many cases it can have the same tragic outcome as with the young boy that did not reach the hospital in time…

Read more about (dangerous) metrics in the Software Industry:
Software Engineering Metrics: What Do They Measure and How Do We Know?, by Cem Kaner.
Metrics, Schmetrics, by Matthew Heusser.
Meaningful Metrics, by Michael Bolton
Measurements/Metrics/Analysis/Judgment, by Rikard Edgren