I know of some testers who are pointed out by others to be Bug Magnets; people recognized for their ability to somehow draw bugs to them. Bug Magnets can be found in many workplaces and I bet that you know of someone that falls under this description. I have been appointed a Bug Magnet by some and it have made me thinking on what this phenomena boils down to.
Is it luck? Is it faith? Is it an ability that some are born with and some aren’t? Can you learn this ability? Can you improve it?
I have wondered about this for some time.

However, this summer I had a revelation when I watched an episode of the marvelous TV series “Homicide: Life on the Streets”:

Homicide: Life on the streets
Season 1 Episode 9 “Night of the dead living”
Det. Frank Pembleton and Det. Tim Bayliss

Bayliss [sits pensively. To Frank]: What are you looking at?

Pembleton is playing cat’s cradle.

Bayliss: You have something that you wanna say to me?

Pembleton: Adena Watson. So many unanswered questions.

Bayliss: And you’re saying that I’m not asking them.

Pembleton: I’m saying that you’re not answering them. [He peers at Tim through the cat's cradle.]

Bayliss [sighs]: What questions aren’t I answering, huh?

Pembleton [gets up and flips through a notebook, draws a picture]: Okay, these sixteen row houses on the north side of Kirk Avenue. Adena’s body was found outside the kitchen door in the red yard at 718 Kirk. Now the killer could have dropped her anywhere. Why not the common alley? Why not the yards at either end of the block? These three row houses are empty. One, two, three. The killer would have stood much less of a chance of being seen if he’d dumped her body in any one of these yards. Why would he risk bringing a little girl’s body inside a closed fence of an occupied house?
Maybe he wanted her body to be found immediately. Maybe he wanted to cast suspicion on the people in 718. Maybe he had some … perverse sense of remorse, some impulse to leave her body inside an enclosed yard to protect her from stray dogs.

Bayliss: These are *exactly* [taps the notebook] the questions that I have been trying to answer.

Pembleton: Well, you can try, but you never will.

Bayliss: Why?

Pembleton: You don’t think like a criminal. You don’t have a criminal’s mind.

Frank walks away. Tim grimaces in disbelief.

Bang! Suddenly it struck me. “You don’t think like a criminal. You don’t have a criminal’s mind”.
In order for a police to think of possible outcomes of a crime they have to be able to think as criminals, and put themselves in a criminal’s way of thinking.
If you don’t do this, you are trying to understand and explain the crime based on your logic.
Similarly, in order to “attract” bugs you have to wanna see problems; you have to identify problems that might bug several kinds of stakeholders; you have to put all your knowledge about the project in to consideration; and you have to be able to see the problems that matter. You are not trying to understand and explain the system by using your own logic, instead you are using several input sources to do this: logic, subjective thoughts, people’s skill levels, complexity of system, technology, etc.

So let me present my take on a definition of “Bug Magnet”:

Being a Bug Magnet = Able to foresee possible problems (or able to spot opportunities for things to go wrong), in context.

The most important thing here is the last two words “in context”. That is, even if you have all the bug taxonomies and oracles in the world to support you, you have to be able to understand what matters in this project.
Knowledge about “all common problems in .NET applications” can help you sometimes. Knowledge about “all common problems in .NET applications that developer X often produce”, is however much more useful.
Understanding what bugs our users is more helpful than knowing about what bugs users in general.
Knowing about problems with focus in Windows applications is one thing, finding these problems during testing is to be able to spot opportunities when they are presented to you in your context (see http://thetesteye.com/blog/2010/11/windows-focus/ ).

Now, back to the questions in the beginning of the post.

• Is it luck?
  – No, even if luck sometimes help. For others it is more of welcoming serendipity.
• Is it faith?
  – I don’t believe in faith.
• Is it an ability that some are born with and some aren’t?
  – Maybe. Some people might have a more developed talent, but I think that most people have this talent. Some are born with variations of narcissistic personality disorders and might have difficulties with this.
• Can you learn this ability?
  – I believe so.
• Can you improve it?
  – Yes. However, you might need to consider one or several dimensions to improve: Empathy, reasoning, attention for detail and seeing the whole, recognizing patterns of your own and other peoples mistakes, subjectivity,  general systems theory, context-driven testing, and more.
  Notice that these dimensions are not technical but rather comes from social sciences.

I do not see myself as a member of any of the Schools of Testing, and I have ethical problems with labelling other people than yourself.
However, I see the schools as a fruitful tool for enhancing your understanding of views on testing.
So please join me in the following thought experiment.

The following is not a perfect match of my personal opinions, it is fiction. I try to be a bit funny and serious at the same time, and I know this is easily misunderstood.

Suppose I run a consultency firm specialized in testing.
We take any kind of testing job (we can also do checking only if requested) and we are very successful; we provide a lot of valuable information to our clients.
The key to our success comes from this recipe:

1. A lot of exploratory testing
ET gives better results, and by explicitly giving testers responsibility and freedom for their activities, they stay motivated and get better over time.
It happens that clients question this approach, but usually it ís sufficient to point at the many Internet resources, and say that they have to be modern, this is the greatest super-good practice right now.

2. Training scheme
We hire people that are fast learners, curious and ambitious. All must take AST courses Foundation, Bug Advocacy, Test Design (the bug reporting is essential, we impress our clients early on, and is the reason we can offer a Money-Back-Guarantee.)
Then everyone must go Rapid Software Testing with Bach or Bolton, which inspires and give breadth to the thinking.
Finally they take Foundation and Advanced ISTQB certification (this is last to avoid problems at previous classes.)
We are no fans of ISTQB, but it is good to know what others know, and we eliminate the risk of losing a deal on a (ridicilous) client requirement.
We also do continuous training on collaboration and feedback, after all, people are the most important part of any context.

3. SBTM
We manage the testing efforts by three sessions a day, this squeezes out maximum from the testers without wearing them out.
We log all the proposed statistics so we can show numbers and progress.
We always add 24% to originally planned sessions, and make sure this cover unexpected things.
Planning and other processes are run context-wise, we use whatever the client is using.

4. Standards
For all jobs, we walk the client through thetesteye’s Quality Characteristics, and a secret company standard for Multiple Information Sources.
The client decides what’s most important in dialogue with us. (We of course also do at least one test per requirement.)
Testers use HICCUPPS(F), CRUCSPIC STMP, CIDTESTD and SFDPOT in their daily work, we know it’s not perfect, but easy to remember.

5. Reasonable Resource Allocation
In a typical project we want to have 1 tester per 3 developers. We tell clients this is our standard where we know we can do a good job. We can do higher or lower ratio, but clearly communicate which expectations this should give.
This makes it easy for clients to measure cost; it has a direct relationship with no. of developers, development time, and start of test effort (the sooner the better, but clients decide.)
Since we have a common background and practices, it is easy to replace our factory workers if needed. But we always keep 1 person from the old staffing, so crucial knowledge can be transferred.
Scaling up or down is not a problem, we have a pool of skilled resources, that grows over time as we hire more talents.

I feel we have a highly manageable process, that repeatedly bring good results. Sure, all projects are unique, but this scheme has proven to be very good, so far.

So what do you think dear readers, would this be a good approach?
Is it Factory, Context-Driven, or something else?

Over the years I have noticed that System Testing have had a special meaning at every place I have been at; and it has even meant different things for people on the same place. I.e. System Testing is depending on the context; and it is fuzzy because we are dealing with arbitrary and/or general systems.

“System testing of software or hardware is testing conducted on a complete, integrated system…” but the boundaries on what the complete and integrated system includes may vary a lot if you look at it from an outside perspective as well as if you look at it from within.

Some examples: When developing a cell phone, where does the “system” begin and where does it end? Does it begin with the cell phone OS or the phone (hardware + OS + software)? Do we need to include any network? How large network? What if you are a third party and developing the phone OS (that could be run standalone in a simulator)? What if you develop an application for the phone (or even different kind of phones)?

So when is a system “complete and integrated”?

I guess that it is very hard to define this because the boundaries of system testing are elastic, and this is true since the boundaries of an arbitrary system are hard to define. Still, we use the term “system testing” daily.

So what is my point really?

I have often found interesting things just by thinking on what system I am working with. I have found out that some things are really outside the boundaries that I first believed were in; and perhaps more useful has been to find things that really should be included in my system and the system testing. And the boundaries might stretch the further you get in a project; what was a complete and fully integrated system during the first half of the project might just be a sub-part at the end of the project.

So my advice is that you take a few minutes now and then in order to seriously ponder the boundaries of Your system. It will affect the type of testing You choose to focus on; and it will be a good aid in order to get a proper and valid test coverage.

In many years I have loved most of what is written by the people behind the context-driven school of testing.
But I have also felt that there is something that isn’t a perfect match.

For a time I thought it was because I saw a few different tries to push people to different schools – which I had ethical problems with – but I realize it is possible to belong to one school, but not push opinions and school membership onto others.

I also had the idea that the school had stopped focusing on people (who aren’t mentioned in What is context-driven testing?), but I realize that was just one blog post, people are still the most important part of the context, see http://www.context-driven-testing.com/

I came a little bit closer when I disliked CAST 2009 theme Serving Our Stakeholders, feeling that testers were diminished and put in a hierarchy of some sort (I would rather focus on the whole team with joint ownership.) But I have realized that serving in this context means giving, not doing what somebody else has ordered.

Finally I have come to the conclusion that the reason why I’m not a context-driven tester is that there are too many contexts where I wouldn’t (wanna) work (which implies a kind of context-imperial testing.)
Examples are situations where testers aren’t allowed to step out of the scripts, or if bugs not stemming from requirements are ignored, or situations where you don’t interact in any ways with other parts of the development, and other departments, or where you only are allowed to be involved for a very short period of time, or where you know that no one wants to create great software, they just want to release at time.

The opposite of these examples are “very good practices for me”, and I totally embrace the exploratory style of testing, and try to use all possible methods and techniques that could be beneficial to the goal I like: world-class software.
As a tester, I don’t feel I can be context-driving, but I have the luxury of choosing which rides to join.

But wait a minute!

If testers can report the current quality status based on the definition above, it means that test cases corresponds to the requirements; and bugs found are violations where the product characteristics does not satisfy the requirements. If so, then you must have requirements that follow a couple of truths:

• Each requirement should exhibit the statements: Correct, Feasible, Necessary, Prioritized, Unambiguous and Verifiable.
• The set of requirements cover all aspects of people needs.
• The set of requirements capture all people expectations.
• The set of requirements corresponds to the different values that people have.
• The set of requirements contains all the different properties that people value.
• The set of requirements are consistent.

(The word People above include: Users, customers, persons, stakeholders, hidden stakeholders, etc.)
At the same time, we know that it is impossible to test everything; you cannot test exhaustively.

But assume, for the sake of argument, that all requirements were true according to the list above; and the testing was really, really extensive; and the test effort was prioritized so that all testing done was necessary and related to the values that the important stakeholders and customers cared about.
If this would be the case, then how can you compare one test case to another? How can you compare two bugs? Is it possible to compare two bugs even if you have 20 grades of severity?

We, as testers, should be subjective; we should do our best to try to put ourselves in other people’s situation; we should find out who the stakeholders are and what they value; we should try to find all problems that matter.
But we should also be careful when we try to report on these matters. And it is not because we haven’t got any clue about the quality of the product, but we should be careful because many times we report on the things that we do that can be quantified and take these as strong indicators of the quality of the product. E.g., number of bugs found, number of test cases run, bugs found per test case, severe bugs found, severe bugs found per test case per week, etc. You know the drill…

If you are using quantitative measurements, you need to figure out what they really mean and how they connect to what really should (or could) be reported.

If you think that “non-technical” people are pleased by getting a couple of digits (hidden in a graph) presented to them, it is like saying: “Since you aren’t a technical person we have translated the words:  Done , Not quite done, Competent, Many, Problems, Requirements, Newly divorced, Few, Fixed, Careless, Test cases, Dyslexic, Needs, Workaholic, Lines of code, Overly complex code, Special configuration, Technical debt, Demands, etc, to some numbers and concealed it all in one graph that shows an aggregate value of the quality”.

I think that it is a bit unfair to the so-called non-technical…

Instead, we should use Jerry Weinberg’s definition “Quality is value to some person” in order to realize that quality is not something easy to quantify. Quality is subjective. Quality is value. Quality relates to some person. Quality is something complex, yet it is intuitive in the eyes of the beholder.

I use Jerry Weinberg’s definition of quality: “Quality is value to some person”; and I use Cem Kaner’s extension to the definition so that it becomes “Quality is value to some person (that matters)”…

I.e. quality is inherently subjective. And there are a lot of persons that are affected by software that we produce… With this in mind it becomes hard for a tester to stay focused when there are so many persons with opinions that could matter; but if we can find out “who matters” we decrease the number of possible values to care about. Still, this will leave us with several important values that need to be taken into account when testing the product.

So how can we testers deal with that?

You could do a role play when testing and put on someone’s hat during the test session; or you could let real users test the product and let them have a say about what they find.
But for a skilled tester it is more about being multidimensionally subjective and think as several persons at the same time.

This means that a lot of values, beliefs and preferences are taken into account which might matter. Not as an average, but as several independent quality dimensions that has (more or less) importance. The hard thing is to know when a value is threatened and for which (type of) person that is affected; and if this matters at all.
I.e., it is a matter of questioning “is there a problem here?” constantly and try to pair a potentially threatened value with its corresponding person. And if this problem threatens a value for some person that matters, we have found a bug. This corresponds to the definition of bug from Cem Kaner “A bug is something that threatens the value of a product”.

Much of this happens automatically for many of you skilled testers out there; when I thought of it recently I realized that this is something I do more and more and hopefully I am improving this skill each day. This is a great skill to have when testing software!

Anyone having any thoughts on this?
Have you experienced this yourself?
If not, does it sound like an interesting thing to examine? Would this be helpful to you?

Cheers,
Henrik

Update 2009-09-14: According to comment from Michael Bolton, see below, the quotes that I said belonged to Cem Kaner are both quotes from James Bach. I apologize for referencing wrong person.

This was originally meant as an answer to the (ironic) thread http://thetesteye.com/blog/2009/06/long-live-the-waterfall/ where a new thread was forked when Ola Janson launched a couple of questions regarding agile development. My answers and thoughts on those questions are listed here.

In one reply to Ola, Rikard says that he has “…never worked in a truly Agile project…” but what is a “truly Agile” project really?

I have worked in both agile (quick and well-coordinated in movement) and Agile (http://www.agilemanifesto.org/) projects.
Here are my thoughts and observations.

• Being agile as a team could apply to any jelled team with a developed group dynamic which manages to quickly respond to problems that might arise. It could also be applicable for any team that has ability to minimise the cost of change instead of trying to avoid the change. I have been in such teams.
• Being Agile as a team is when the team agrees upon the Agile Manifesto (Note: remember that the Agile Manifesto is a set of values and not a set of laws). I have been in such teams.
• XP, Scrum, DSDM, Lean, Kanban, etc are process tools in that they help you work more effectively by, to a certain extent, telling you what to do. These process tools must live up to the Agile Manifesto and are described practises for how to work in an Agile fashion more or less structured; each method has its benefits and drawbacks and none of them are comprehensive. That is why you often see combinations of different “Agile” processes e.g., XP (developing method) and Scrum (team management method) and Continuous Integration (incremental build method).
• You can work in an agile team and still utilize Agile methods such as Scrum or XP; but you could also use other non-Agile methods and still be agile.
• You can work in an Agile team and don’t utilize any of the big recognized “Agile” methods.
• Being a tester in an agile team (or project) is gold.
• Being a tester in an Agile team (or project) can be a smooth experience but it could also be painful. My experience is that if you as a tester are not engaged or involved in the team you will have difficulties. E.g., since documentation is not prioritised you need to know more about the software continuously and therefore there will be problems if you are not engaged in the day-to-day development of the project. And many times the team has been forced into working according to a specific Agile method that does not seem to be suitable for the context of project; and it might be very well suited for programmers but not at all suitable for the testing tasks.

The key thing with the agile movement, as I see it, is that the methods on how to develop software have evolved from the project context and, perhaps most importantly, being defined by the team members themselves. Therefore I believe that those projects have had a successful outcome, at least from the team-perspective. Many times it has correlated with business success.

The Agile movement on the other hand, which originally has sprung from the agile movement, has become more and more strict and promoting “best practises” on how to work Agile. This has become the new cash cow for many consultant firms and they of course promote and teach the “real way” of doing things. But I guess that many founders of the Agile Manifesto really think that it should be more agile than Agile…
So nowadays there are, and will be more and more, voices raised which question the new face of agile/Agile and instead promote the “original” thoughts; or at least the thoughts that they believed in then and still do.

My conclusion is: Find out what matters for you; your team; and project. Define a process that will work for you.

Read more about Agile vs. agile:

• Agile Versus agile Development - http://www.agilejournal.com/articles/columns/from-the-editor-mainmenu-45/18-agile-versus-agile-development
• Defining Agile Methodology - http://www.satisfice.com/blog/archives/45

Find out if your team is agile (or is it Agile?)

• Back-of-a-Napkin Agile Assessment - http://testobsessed.com/2008/11/28/back-of-a-napkin-agile-assessment/

Examples of reactions on current Agile methods:

• Artisanal Retro-Futurism crossed with Team-Scale Anarcho-Syndicalism - http://arxta.net/
• Cargo Cult Agile – http://www.exotribe.com/node/16
• WAgile - http://www.parlezuml.com/blog/?postid=708
• Who Stole Agile? - http://www.satisfice.com/blog/archives/51

Group dynamics

• Read about “Jelled team” (and other interesting topics) in Peopleware - http://www.dorsethouse.com/books/pw.html

Well, “secretly” as in that he does not know that I am in love with him… Yet!

If you haven’t discovered the amazing Cem Kaner yet, I can give you the following advices and hoping that you too might fall in love some day:

• Visit kaner.com publications and read ANY article from his large publication-section.
• Buy and read any of his fine books (I would especially recommend Lessons Learned in Software Testing)
• Follow his blog on http://www.satisfice.com/kaner/ (though he updates too rarely)!
• If you haven’t got the possibility to see him lecture in real life, take the BBST online course! http://www.testingeducation.org/BBST/index.html
  And yes, it is for free!
• Read his words about Context-Driven Testing and what it mean; and what it doesn’t mean: http://www.context-driven-testing.com/
• Join the context-driven testing forum and read his stunningly thoughtful comments: http://groups.yahoo.com/group/software-testing/

Are you still not convinced about his greatness, I urge you to talk to any test expert in our industry and see what they have to say about him.
Note: A lot of work above is collaborations with other great people in our industry e.g., James Bach, but they don’t really touch me as much as Cem do.

Recently in Sweden there was a tragic death to a young child that could have been rescued if only the child had come to a hospital in time for a full exam. The one that was blamed for this death was the medical care hotline company that did not understand the severity of the illness and did not send this kid to the hospital. (Read more, in Swedish: http://www.dn.se/sthlm/pojke-dog-efter-rad-att-inte-aka-till-sjukhuset-1.859096)

After this tragic accident, it was discovered that this private medical care hotline company pays out a monthly bonus to those nurses that keep their phone calls short. (Read more, in Swedish: http://www.dn.se/sthlm/skoterskor-far-bonus-for-snabba-rad-1.864534 )
I.e. if they keep the call below 3.48 minutes and during that time complete the medical record, they receive a bonus of 1000 Swedish kronor (approx. € 100). In order to receive the bonus, there are some quality goals as well. E.g., you don’t get the bonus if you unnecessarily send someone to the emergency ward; or if you give a faulty medical advice.
Do I need to tell you that the county council paid the private company by the number of calls they handled.

This is what happens when you use simplified and dangerous metrics as a foundation for incentive pay… And these metrics are easy to abuse because they are based on simplified models of how the real world looks like.
When dealing with people, you are dealing with “complex systems” (read more in An Introduction to General Systems Thinking, by Gerald M. Weinberg ) and you cannot treat every person like they would be the same. I.e., the people calling in (and indeed children that cannot speak for themselves) are treated as a neutral “* 1” or “+ 0” in the metrics equation.
This happens if you include simplified metrics to measure your efficiency when dealing with people; metrics that leaves out the most important and complex parts of the equation: humans and human interaction.
Nurses know how to work with people, they know that people are unique; they know that their job is hard and requires skill and years of experience. They know that some patients require 20 minutes before they are calm or they need such time to explain everything important; they also know that some people just need 25 seconds before they are satisfied.
It is a shame that nurses are measured by how fast they finish a phone call.

It is the same thing that happens again and again in software industry; or rather the peopleware industry. People that work with developing software are measured by metrics that are dangerous and wrong; and in many cases it can have the same tragic outcome as with the young boy that did not reach the hospital in time…

Read more about (dangerous) metrics in the Software Industry:
Software Engineering Metrics: What Do They Measure and How Do We Know?, by Cem Kaner.
Metrics, Schmetrics, by Matthew Heusser.
Meaningful Metrics, by Michael Bolton
Measurements/Metrics/Analysis/Judgment, by Rikard Edgren

This was originally a response to Rikard’s post “Multi-Dimensional Software Testing”, but here I have developed my thoughts a bit.

As I see it, there are more or less obvious stakeholders and stakeholders that might be more or less hidden. A “customer” might be such an obvious stakeholder. It might then just be a matter of recognizing which of these “customers” that matter and take those expectations into consideration when selecting test activities.

The project manager is another example of an obvious stakeholder.

And a product owner would naturally be considered an obvious stakeholder.

But how about the hidden stakeholders; and what are their interests in the project/team?

Maybe the members of the test team would have interests in the team’s work that would affect the testing activities (i.e. their own work)? Someone in the team might be dependent on other peoples work. Some people in the team might lack important skills which would affect the chosen test activities. Are there some people that only can work part time and thereby affect which times meetings are held in the team?

The members of the test team are them self stakeholders to the project and might have important interests. This might have impact on the testability, documentation effort, release schedules, communication, etc. Project members other than the testers are also stakeholders to the project and they might have their specific interests.

Business managers might have much to say about the testing activities; and they might especially be interested in the cost of the testing activities. They might have influence on hardware or software that would be needed and thereby affect the testing activities.

Other test groups that work in parallel might be interested in the outcome; and especially to make sure that your team doesn’t over-deliver and make their team look bad. And they might be interested in what type of cooperation that could take place in order to be more effective. They might also have an interest in what type of reporting to do in order to compile a common test report. Other test groups might also be interested in creating a common vocabulary to use in the test scripts.

The maintenance team that would receive the product after it has been released might be interested in the outcome. And especially they might be interested in the format of the test product i.e. documentation, test scripts, procedures, etc. They might also be interested in good practises that have come up during the project; and if these good practises were designed to work for the maintenance team, they have indeed already affected the testing activities in your team.

Certainly there are more hidden stakeholders than those above to take into consideration when selecting test activities or test strategy to focus on. But my goal is to illustrate the problem with a couple of hidden stakeholders in order to make you think about your situation and possible stakeholders that might matter for you.

Note:
Many of these stakeholders are already taken into consideration on a daily basis, but I believe that they are not considered as stakeholders. Maybe you would discover more important aspects of each stakeholder if you tried to analyse each of them more thoroughly.

E.g., try to list all people or roles that matter to the team and start to analyse what their interests are and how they might affect your team/project.

And you should not be surprised that some of these were hidden to you at first, and when their interests are revealed you might discover that they would have a large impact on your testing activities.