I know of some testers who are pointed out by others to be Bug Magnets; people recognized for their ability to somehow draw bugs to them. Bug Magnets can be found in many workplaces and I bet that you know of someone that falls under this description. I have been appointed a Bug Magnet by some and it have made me thinking on what this phenomena boils down to.
Is it luck? Is it faith? Is it an ability that some are born with and some aren’t? Can you learn this ability? Can you improve it?
I have wondered about this for some time.

However, this summer I had a revelation when I watched an episode of the marvelous TV series “Homicide: Life on the Streets”:

Homicide: Life on the streets
Season 1 Episode 9 “Night of the dead living”
Det. Frank Pembleton and Det. Tim Bayliss

Bayliss [sits pensively. To Frank]: What are you looking at?

Pembleton is playing cat’s cradle.

Bayliss: You have something that you wanna say to me?

Pembleton: Adena Watson. So many unanswered questions.

Bayliss: And you’re saying that I’m not asking them.

Pembleton: I’m saying that you’re not answering them. [He peers at Tim through the cat's cradle.]

Bayliss [sighs]: What questions aren’t I answering, huh?

Pembleton [gets up and flips through a notebook, draws a picture]: Okay, these sixteen row houses on the north side of Kirk Avenue. Adena’s body was found outside the kitchen door in the red yard at 718 Kirk. Now the killer could have dropped her anywhere. Why not the common alley? Why not the yards at either end of the block? These three row houses are empty. One, two, three. The killer would have stood much less of a chance of being seen if he’d dumped her body in any one of these yards. Why would he risk bringing a little girl’s body inside a closed fence of an occupied house?
Maybe he wanted her body to be found immediately. Maybe he wanted to cast suspicion on the people in 718. Maybe he had some … perverse sense of remorse, some impulse to leave her body inside an enclosed yard to protect her from stray dogs.

Bayliss: These are *exactly* [taps the notebook] the questions that I have been trying to answer.

Pembleton: Well, you can try, but you never will.

Bayliss: Why?

Pembleton: You don’t think like a criminal. You don’t have a criminal’s mind.

Frank walks away. Tim grimaces in disbelief.

Bang! Suddenly it struck me. “You don’t think like a criminal. You don’t have a criminal’s mind”.
In order for a police to think of possible outcomes of a crime they have to be able to think as criminals, and put themselves in a criminal’s way of thinking.
If you don’t do this, you are trying to understand and explain the crime based on your logic.
Similarly, in order to “attract” bugs you have to wanna see problems; you have to identify problems that might bug several kinds of stakeholders; you have to put all your knowledge about the project in to consideration; and you have to be able to see the problems that matter. You are not trying to understand and explain the system by using your own logic, instead you are using several input sources to do this: logic, subjective thoughts, people’s skill levels, complexity of system, technology, etc.

So let me present my take on a definition of “Bug Magnet”:

Being a Bug Magnet = Able to foresee possible problems (or able to spot opportunities for things to go wrong), in context.

The most important thing here is the last two words “in context”. That is, even if you have all the bug taxonomies and oracles in the world to support you, you have to be able to understand what matters in this project.
Knowledge about “all common problems in .NET applications” can help you sometimes. Knowledge about “all common problems in .NET applications that developer X often produce”, is however much more useful.
Understanding what bugs our users is more helpful than knowing about what bugs users in general.
Knowing about problems with focus in Windows applications is one thing, finding these problems during testing is to be able to spot opportunities when they are presented to you in your context (see http://thetesteye.com/blog/2010/11/windows-focus/ ).

Now, back to the questions in the beginning of the post.

• Is it luck?
  – No, even if luck sometimes help. For others it is more of welcoming serendipity.
• Is it faith?
  – I don’t believe in faith.
• Is it an ability that some are born with and some aren’t?
  – Maybe. Some people might have a more developed talent, but I think that most people have this talent. Some are born with variations of narcissistic personality disorders and might have difficulties with this.
• Can you learn this ability?
  – I believe so.
• Can you improve it?
  – Yes. However, you might need to consider one or several dimensions to improve: Empathy, reasoning, attention for detail and seeing the whole, recognizing patterns of your own and other peoples mistakes, subjectivity,  general systems theory, context-driven testing, and more.
  Notice that these dimensions are not technical but rather comes from social sciences.

Over the years, one has read quite some text about software testing.
Some things have (from various sources and experiences) become clear for me, and I’m surprised when seeing articles/presentations that don’t acknowledge these insights.
These “truths” are now and then implicitly disregarded:

* Requirements don’t include all important information
* Testing includes more than verifying explicit requirements
* Testing is only partly about measuring and verification
* Any tester can perform non-functional testing (to some degree)
* Non-functional testing is efficient to perform at the same time as integration/system testing
* Metrics (measurement + value) are dangerous
* Quantitative numbers omit important information from the qualitative results
* Subjectivity is something good
* Code coverage has little meaning
* Test automation is primarily for regression testing, which often isn’t the biggest part of the testing challenge
* Software development & testing is complex, and analogies from manufacturing are not good
* Testers should collaborate with developers and others
* We don’t have complete knowledge, so exploring is necessary

Correct me if I’m wrong!

I stopped myself in the hallway just to ask a few quick questions:

Why do you like software testing?
- That’s not a short question! But it involves a lot of creativity, subjectivity, serendipity, critical and holistic thinking.

What’s your motivation for writing on a blog?
- I don’t know, I just can’t stop myself.

How come?
- Not sure, I think I came to a place where writing was the best way to learn things; and learning things is an absolute must.

What’s your favorite quote, right now?
- You can see a lot by just looking (Yogi Berra)

Any new paradoxes?
- No, just an old one: With check lists you do testing, with test scripts you do checking.

Who is the best tester?
- Can’t tell, and if I could, I wouldn’t. Testing is part of a team effort to develop software; it’s not a competition.

Which one of Kaner’s 101 coverages do you prefer?
- A broad version of #89: Potential Usage Coverage.

What bug would you like to find?
- A very important one that happens after very normal actions, but in a long, credible scenario. It should be on a non-released system, so it can be fixed before causing real problems.

What’s the recipe for Good Software Testing?
- 1 part preparations, 2 parts knowledge, 3 parts wide-open, trained senses, 4 parts hard work, and a big pinch each of creativity and subjectivity.

How many testers actually spend more than 5% of their time testing non-functionals?
- I think many have quality characteristics in the back of their head all the time; so for me it is 100% of the testing time.

Can you say something positive about ISTQB Certification?
- In the Advanced Syllabus there is an appendix with a lot of good recommendations regarding pitfalls in software testing.

What’s the most difficult virtue for a tester?
- Humility.

Can you think of any strange bugs?
- A crash bug when cancelling a dialog, but only if the dialog had been manually moved.

Nothing better than that?
- A crash when a value range dealt with 9.8922, 9.975, 10?

What’s your favorite accelerator?
- Windows+Break. Everybody doesn’t seem to know that this is the fastest way to bring up the System details.

If software testing is a social science, which qualitative methodology can we learn most from?
- Parts of Grounded Theory is a perfect fit.

What’s your favorite string to use when testing?
- “This is a tribute to the longest string in the world.”

What is the danger of solving problems?
- It wasn’t the important thing, and now you have stopped looking.

Do you have a lesson for software testing?
- No, but the last one in the book Lessons Learned in Software Testing is excellent, the one on Test Strategy Heuristics.

Do you want to be Deming or Fleming?
- Ah, you’re thinking serendipity-wise. Fleming and the penicillin, of course.

Do you have an alternative to SMART goals?
- I want my goals to be Vague, Ongoing, Motivating, Important and Trustworthy.

Where would you like to have an extra pair of eyes?
- I just have to steal this answer: I would give them to a blind.

How long does it take to become a really good tester?
- Don’t know. I might be there, and if so it happened gradually over many years. But it takes about one year to master sourdough baking, so it could be something similar for testing, if you have the basic skills, knowledge and motivation that are needed (don’t ask me about that!)

What’s your biggest weakness?
- Could be that I am easy to misunderstand. I often try to be serious and humuorous, at the same time; which can be very difficult to evaluate.

What development process are you using at your current assignment?
- Same as everybody else, a mix of Waterfall and Agile.

What do you have against Agile?
- Nothing really. It’s just so over-used, and doesn’t seem to mean much more than it’s something good (just like “professional”)

Don’t you agree that testing is a service?
- Can’t say it’s not true (and it’s extra important for testers to be service-minded), but I would talk about long-term team ownership several times before using service vocabulary.

If you were a young, aspiring software tester with a lot of spare time, what would you do?
- I would find an open-source tool I like (maybe a test tool?), and test it thoroughly, learning everything I can about the test approaches while using them.

Are you a certified tester?
- No, but I have a diploma from a basic software testing course.

What’s in your Software Testing Dystopia?
- Extreme specialization, same work all the time, no system overview, and cumbersome software with just some value.

What questions would you not want to be asked?
- What’s the most important aspect of software testing?

Anything else you would like to add?
- I am always on the testers’ side!

Thank you very much! Have a good day, and a nice vacation.

I have been thinking about qualitative research lately, and wondered what the question(s) would look like if testing was seen as a research project.

The testing effort has many positive effects, but one common and important is to provide information about the product, so a good release decision can be made. We cannot prove that the software is good, but we can try our very best to show important ways it can fail.
But at the same time, it is too late to come with important information just before the release is to be made; the information should be provided in a timely manner.
So in a paradoxal way, we try to destroy the product we love, but at the same time give the information to the project, so they can make the testing mission fail.

Testing try to answer “No” to the “Can we ship?” question – many times in many ways – but gives the project the best chances to resolve all issues as early as possible.

Another important question is “How can the product be better?”, and information about this is also being provided all the time. Not too late, but also not too early, in times when we know too little, and the cost is too high to reach the vital information.

Of course, there are many other sources and considerations, but testing often has the best view of the system as a whole, and the best view of some details from a customer perspective.
Testing can’t be complete, but the answers are valid when the testing can be considered saturated, when too much effort is needed to get new information that aren’t too important.

Are there other questions that are more important?
Or are we providing answers without questions?

Software Potato

 The square symbolizes the features and bugs you will find with test cases stemming from requirements (that can’t and shouldn’t be complete.)
The blue area is all bugs, including things that maybe no customers would consider annoying.
The brown area is all important bugs, those bugs you’d want to find and fix.

So how do you go from the requirements to “all important bugs”?
You won’t have time to create test scripts for “everything”.
So maybe you do exploratory testing (thin lines in many directions), and hope for the best.
Or maybe you test around one-liners (thicker horizontal lines), that are more distinct, that are reviewed, and have a better chance of finding what’s important.
Either option, some part luck, and a large portion of hard work is needed.
But I think you have a much better chance if you are using one-liners, especially if it’s a larger project.

Later I have realized that one-liners aren’t essential; that this problem has been solved many times at many places with many different approaches.
What is common could be that testers learn a lot of things from many different sources, combine things, think critically and design tests (in advance or on-the-fly) that will cover the important areas.
Maybe we need a name for this method; it could be Grounded Test Design.

There are some things about software testing that are special, but not unique:
* you are never done, and there is always something to do
* you have to be creative very often
* you are dependent on new, different and conflicting technologies, users, objectives
It’s not easy to be a tester, thank God for that!

And there are some things that are unique:
* it is often good to do things in the wrong way
* you try to destroy something you love

You might say that none of the above is true for an executor of detailed test cases, but that’s not software testing, it is software checking!

I just want to promote a really good blog post written by Michael Bolton where he describes the difference between Testing and Checking:
http://www.developsense.com/2009/08/testing-vs-checking.html

I wish that many managers, testers and developers read this post…

Cheers,
Henrik

Well, “secretly” as in that he does not know that I am in love with him… Yet!

If you haven’t discovered the amazing Cem Kaner yet, I can give you the following advices and hoping that you too might fall in love some day:

• Visit kaner.com publications and read ANY article from his large publication-section.
• Buy and read any of his fine books (I would especially recommend Lessons Learned in Software Testing)
• Follow his blog on http://www.satisfice.com/kaner/ (though he updates too rarely)!
• If you haven’t got the possibility to see him lecture in real life, take the BBST online course! http://www.testingeducation.org/BBST/
  And yes, it is for free!
• Read his words about Context-Driven Testing and what it mean; and what it doesn’t mean: http://www.context-driven-testing.com/
• Join the context-driven testing forum and read his stunningly thoughtful comments: http://groups.yahoo.com/group/software-testing/

Are you still not convinced about his greatness, I urge you to talk to any test expert in our industry and see what they have to say about him.
Note: A lot of work above is collaborations with other great people in our industry e.g., James Bach, but they don’t really touch me as much as Cem do.

Thare are many ways to look at the problem of testing software, and it is rarely wise to use only one; that’s why there are so many mnemonics, e.g. SFDPOT, CRUSSPIC STMPL, HICCUPPS, FCC CUTS VIDS (http://www.satisfice.com/tools/satisfice-tsm-4p.pdf, http://www.developsense.com/articles/2005-01-TestingWithoutAMap.pdf, http://www.testingreflections.com/node/view/2823)
Here are some questions for your confusion:

1) What?
a) The functionality, what the software does from A to Z; what elements the software works with; the source code, from beginning to end; the requirements, the use cases, the test cases; the product as a whole, including documentation and support. None of these give the full picture.
b) The actual tests, their artifacts, the communication.

2) How?
a) The attributes of the functionality; Usability, Performance, Security, Reliability, Installability, Compatibility, Scalability, Accessibility.
b) The process (both the documented and the real) used when testing; your techniques and approaches, your tools.

3) Who?
a) The (potential) users, which you probably can’t bring down to a couple of personas, but it is good to know what the users are like; what language they speak; what most of them knows.
b) The testers, your (different) skills. Will the developers also do testing? Will customers test the software, in order to accept the product or find bugs?

4) Why?
a) What do you think the users are trying to accomplish with the software? Can the software be used in many more number of ways as there are users? Are there any unintentional effects that are desirable? Why should the user not switch to your competititor?
b) What are the main drivers for the testing effort: Find all relevant bugs; provide enough information for the project to succeed; minimize customer complaints; learn and explore?

5) Where?
a) The environments where the software will operate and interact. Specific hardware and software?
b) Which environments can you afford to use; how far can you get with Basic Configuration Matrix (http://thetesteye.com/blog/2008/05/bcm-basic-configuration-matrix/)

6) When?
a) Will the software only be used at one given time, at one place? Will upcoming releases replace the old version, or is it necessary to be forward compatible with currently non-existant software?
b) When will you test the software, and in which shape will it be?

A project manager, that had no knowledge about testing whatsoever, asked me how he should go about testing? The question was vague since he did not know where to start. I wrote him a quick email listing a few things to consider:

• Create a list of all use cases. For each use case consider possible errors. Identify how these use cases and the errors from these could be tested and verified.
• Create a list of techniques used. Identify common errors for the specific techniques. Identify issues that affect these techniques in one way or another. These issues and common errors can be used as a base for testing.
• Create a list of the major areas of functionality that need to be tested. Create a matrix and categorise the areas in two columns (column one: risk anything goes wrong, column two: cost if anything goes wrong) from 1 to 5, where 1 is most costly or highest risk. Sort the categories.
• Identify important milestones when there are deliverables for testers.
• Identify what deliverables you expect from testing. This can be test reports, bugs. Consider when and how often you want these. Be reminded that all administration will take time from regular testing.
• Which test resources are available and to what percentage. Investigate when they are away and if they have other tasks that they work with.
• What testing has been done so far? What is documented? What did they find?
• Is there a backlog of bugs? Are they well tended and classified? Can they be reused and are they still valid?
• Decide on how you want to work with bugs and enhancements
• Have the testers received all possible documentation needed to start working? This can be support documents, help-files, design/functional specifications and so on. These documents can be used for test ideas.
• What will the test environment look like? Will it be a virtual one or a full customer-like environment with hardware and software?
• Have you considered how many prototypes that should be available for the testers? Consider that many might break. Testers without something to test is a no-no.
• Is there any other configurations that we should consider that is outside the test environment that these builds/prototypes need to run in?

If your testers are new to testing consider this:

• Use the requirements to create user scenarios that in turn can create more test ideas.
• Some tests might need to be planned long before they are executed. Example of this can be EMC testing or Safety testing, that might be performed externally in other test labs.
• System testing should be done by testers, thus not a developer or product manager.
• Do not test everything. Focus on costs and risks first. All areas that go untested should be a calculated risk done by stakeholders or decision makers.
• Test everything. All major functionality should have at least one test. Important areas should have deeper testing while less important can have shallow testing.
• Consider long term investments in testing. Will there be more versions of the product that need even more testing. It might be good to invest in test automation if it gives a ROI. Should test cases and test result be saved for the future, in most cases you would like that.

All of this information is usually structured in the various documents that expert testers are used to such as test plans, test strategies and so on. But for someone that did not know anything about testing I just gave him this.

I got no response from the project manager on this though, so either it was total crap or I scared him away.