I stopped myself in the hallway just to ask a few quick questions:

Why do you like software testing?
- That’s not a short question! But it involves a lot of creativity, subjectivity, serendipity, critical and holistic thinking.

What’s your motivation for writing on a blog?
- I don’t know, I just can’t stop myself.

How come?
- Not sure, I think I came to a place where writing was the best way to learn things; and learning things is an absolute must.

What’s your favorite quote, right now?
- You can see a lot by just looking (Yogi Berra)

Any new paradoxes?
- No, just an old one: With check lists you do testing, with test scripts you do checking.

Who is the best tester?
- Can’t tell, and if I could, I wouldn’t. Testing is part of a team effort to develop software; it’s not a competition.

Which one of Kaner’s 101 coverages do you prefer?
- A broad version of #89: Potential Usage Coverage.

What bug would you like to find?
- A very important one that happens after very normal actions, but in a long, credible scenario. It should be on a non-released system, so it can be fixed before causing real problems.

What’s the recipe for Good Software Testing?
- 1 part preparations, 2 parts knowledge, 3 parts wide-open, trained senses, 4 parts hard work, and a big pinch each of creativity and subjectivity.

How many testers actually spend more than 5% of their time testing non-functionals?
- I think many have quality characteristics in the back of their head all the time; so for me it is 100% of the testing time.

Can you say something positive about ISTQB Certification?
- In the Advanced Syllabus there is an appendix with a lot of good recommendations regarding pitfalls in software testing.

What’s the most difficult virtue for a tester?
- Humility.

Can you think of any strange bugs?
- A crash bug when cancelling a dialog, but only if the dialog had been manually moved.

Nothing better than that?
- A crash when a value range dealt with 9.8922, 9.975, 10?

What’s your favorite accelerator?
- Windows+Break. Everybody doesn’t seem to know that this is the fastest way to bring up the System details.

If software testing is a social science, which qualitative methodology can we learn most from?
- Parts of Grounded Theory is a perfect fit.

What’s your favorite string to use when testing?
- “This is a tribute to the longest string in the world.”

What is the danger of solving problems?
- It wasn’t the important thing, and now you have stopped looking.

Do you have a lesson for software testing?
- No, but the last one in the book Lessons Learned in Software Testing is excellent, the one on Test Strategy Heuristics.

Do you want to be Deming or Fleming?
- Ah, you’re thinking serendipity-wise. Fleming and the penicillin, of course.

Do you have an alternative to SMART goals?
- I want my goals to be Vague, Ongoing, Motivating, Important and Trustworthy.

Where would you like to have an extra pair of eyes?
- I just have to steal this answer: I would give them to a blind.

How long does it take to become a really good tester?
- Don’t know. I might be there, and if so it happened gradually over many years. But it takes about one year to master sourdough baking, so it could be something similar for testing, if you have the basic skills, knowledge and motivation that are needed (don’t ask me about that!)

What’s your biggest weakness?
- Could be that I am easy to misunderstand. I often try to be serious and humuorous, at the same time; which can be very difficult to evaluate.

What development process are you using at your current assignment?
- Same as everybody else, a mix of Waterfall and Agile.

What do you have against Agile?
- Nothing really. It’s just so over-used, and doesn’t seem to mean much more than it’s something good (just like “professional”)

Don’t you agree that testing is a service?
- Can’t say it’s not true (and it’s extra important for testers to be service-minded), but I would talk about long-term team ownership several times before using service vocabulary.

If you were a young, aspiring software tester with a lot of spare time, what would you do?
- I would find an open-source tool I like (maybe a test tool?), and test it thoroughly, learning everything I can about the test approaches while using them.

Are you a certified tester?
- No, but I have a diploma from a basic software testing course.

What’s in your Software Testing Dystopia?
- Extreme specialization, same work all the time, no system overview, and cumbersome software with just some value.

What questions would you not want to be asked?
- What’s the most important aspect of software testing?

Anything else you would like to add?
- I am always on the testers’ side!

Thank you very much! Have a good day, and a nice vacation.

I have been thinking about qualitative research lately, and wondered what the question(s) would look like if testing was seen as a research project.

The testing effort has many positive effects, but one common and important is to provide information about the product, so a good release decision can be made. We cannot prove that the software is good, but we can try our very best to show important ways it can fail.
But at the same time, it is too late to come with important information just before the release is to be made; the information should be provided in a timely manner.
So in a paradoxal way, we try to destroy the product we love, but at the same time give the information to the project, so they can make the testing mission fail.

Testing try to answer “No” to the “Can we ship?” question – many times in many ways – but gives the project the best chances to resolve all issues as early as possible.

Another important question is “How can the product be better?”, and information about this is also being provided all the time. Not too late, but also not too early, in times when we know too little, and the cost is too high to reach the vital information.

Of course, there are many other sources and considerations, but testing often has the best view of the system as a whole, and the best view of some details from a customer perspective.
Testing can’t be complete, but the answers are valid when the testing can be considered saturated, when too much effort is needed to get new information that aren’t too important.

Are there other questions that are more important?
Or are we providing answers without questions?

Software Potato

 The square symbolizes the features and bugs you will find with test cases stemming from requirements (that can’t and shouldn’t be complete.)
The blue area is all bugs, including things that maybe no customers would consider annoying.
The brown area is all important bugs, those bugs you’d want to find and fix.

So how do you go from the requirements to “all important bugs”?
You won’t have time to create test scripts for “everything”.
So maybe you do exploratory testing (thin lines in many directions), and hope for the best.
Or maybe you test around one-liners (thicker horizontal lines), that are more distinct, that are reviewed, and have a better chance of finding what’s important.
Either option, some part luck, and a large portion of hard work is needed.
But I think you have a much better chance if you are using one-liners, especially if it’s a larger project.

Later I have realized that one-liners aren’t essential; that this problem has been solved many times at many places with many different approaches.
What is common could be that testers learn a lot of things from many different sources, combine things, think critically and design tests (in advance or on-the-fly) that will cover the important areas.
Maybe we need a name for this method; it could be Grounded Test Design.

There are some things about software testing that are special, but not unique:
* you are never done, and there is always something to do
* you have to be creative very often
* you are dependent on new, different and conflicting technologies, users, objectives
It’s not easy to be a tester, thank God for that!

And there are some things that are unique:
* it is often good to do things in the wrong way
* you try to destroy something you love

You might say that none of the above is true for an executor of detailed test cases, but that’s not software testing, it is software checking!

I just want to promote a really good blog post written by Michael Bolton where he describes the difference between Testing and Checking:
http://www.developsense.com/2009/08/testing-vs-checking.html

I wish that many managers, testers and developers read this post…

Cheers,
Henrik

Well, “secretly” as in that he does not know that I am in love with him… Yet!

If you haven’t discovered the amazing Cem Kaner yet, I can give you the following advices and hoping that you too might fall in love some day:

• Visit kaner.com publications and read ANY article from his large publication-section.
• Buy and read any of his fine books (I would especially recommend Lessons Learned in Software Testing)
• Follow his blog on http://www.satisfice.com/kaner/ (though he updates too rarely)!
• If you haven’t got the possibility to see him lecture in real life, take the BBST online course! http://www.testingeducation.org/BBST/index.html
  And yes, it is for free!
• Read his words about Context-Driven Testing and what it mean; and what it doesn’t mean: http://www.context-driven-testing.com/
• Join the context-driven testing forum and read his stunningly thoughtful comments: http://groups.yahoo.com/group/software-testing/

Are you still not convinced about his greatness, I urge you to talk to any test expert in our industry and see what they have to say about him.
Note: A lot of work above is collaborations with other great people in our industry e.g., James Bach, but they don’t really touch me as much as Cem do.

Thare are many ways to look at the problem of testing software, and it is rarely wise to use only one; that’s why there are so many mnemonics, e.g. SFDPOT, CRUSSPIC STMPL, HICCUPPS, FCC CUTS VIDS (http://www.satisfice.com/tools/satisfice-tsm-4p.pdf, http://www.developsense.com/articles/2005-01-TestingWithoutAMap.pdf, http://www.testingreflections.com/node/view/2823)
Here are some questions for your confusion:

1) What?
a) The functionality, what the software does from A to Z; what elements the software works with; the source code, from beginning to end; the requirements, the use cases, the test cases; the product as a whole, including documentation and support. None of these give the full picture.
b) The actual tests, their artifacts, the communication.

2) How?
a) The attributes of the functionality; Usability, Performance, Security, Reliability, Installability, Compatibility, Scalability, Accessibility.
b) The process (both the documented and the real) used when testing; your techniques and approaches, your tools.

3) Who?
a) The (potential) users, which you probably can’t bring down to a couple of personas, but it is good to know what the users are like; what language they speak; what most of them knows.
b) The testers, your (different) skills. Will the developers also do testing? Will customers test the software, in order to accept the product or find bugs?

4) Why?
a) What do you think the users are trying to accomplish with the software? Can the software be used in many more number of ways as there are users? Are there any unintentional effects that are desirable? Why should the user not switch to your competititor?
b) What are the main drivers for the testing effort: Find all relevant bugs; provide enough information for the project to succeed; minimize customer complaints; learn and explore?

5) Where?
a) The environments where the software will operate and interact. Specific hardware and software?
b) Which environments can you afford to use; how far can you get with Basic Configuration Matrix (http://thetesteye.com/blog/2008/05/bcm-basic-configuration-matrix/)

6) When?
a) Will the software only be used at one given time, at one place? Will upcoming releases replace the old version, or is it necessary to be forward compatible with currently non-existant software?
b) When will you test the software, and in which shape will it be?

A project manager, that had no knowledge about testing whatsoever, asked me how he should go about testing? The question was vague since he did not know where to start. I wrote him a quick email listing a few things to consider:

• Create a list of all use cases. For each use case consider possible errors. Identify how these use cases and the errors from these could be tested and verified.
• Create a list of techniques used. Identify common errors for the specific techniques. Identify issues that affect these techniques in one way or another. These issues and common errors can be used as a base for testing.
• Create a list of the major areas of functionality that need to be tested. Create a matrix and categorise the areas in two columns (column one: risk anything goes wrong, column two: cost if anything goes wrong) from 1 to 5, where 1 is most costly or highest risk. Sort the categories.
• Identify important milestones when there are deliverables for testers.
• Identify what deliverables you expect from testing. This can be test reports, bugs. Consider when and how often you want these. Be reminded that all administration will take time from regular testing.
• Which test resources are available and to what percentage. Investigate when they are away and if they have other tasks that they work with.
• What testing has been done so far? What is documented? What did they find?
• Is there a backlog of bugs? Are they well tended and classified? Can they be reused and are they still valid?
• Decide on how you want to work with bugs and enhancements
• Have the testers received all possible documentation needed to start working? This can be support documents, help-files, design/functional specifications and so on. These documents can be used for test ideas.
• What will the test environment look like? Will it be a virtual one or a full customer-like environment with hardware and software?
• Have you considered how many prototypes that should be available for the testers? Consider that many might break. Testers without something to test is a no-no.
• Is there any other configurations that we should consider that is outside the test environment that these builds/prototypes need to run in?

If your testers are new to testing consider this:

• Use the requirements to create user scenarios that in turn can create more test ideas.
• Some tests might need to be planned long before they are executed. Example of this can be EMC testing or Safety testing, that might be performed externally in other test labs.
• System testing should be done by testers, thus not a developer or product manager.
• Do not test everything. Focus on costs and risks first. All areas that go untested should be a calculated risk done by stakeholders or decision makers.
• Test everything. All major functionality should have at least one test. Important areas should have deeper testing while less important can have shallow testing.
• Consider long term investments in testing. Will there be more versions of the product that need even more testing. It might be good to invest in test automation if it gives a ROI. Should test cases and test result be saved for the future, in most cases you would like that.

All of this information is usually structured in the various documents that expert testers are used to such as test plans, test strategies and so on. But for someone that did not know anything about testing I just gave him this.

I got no response from the project manager on this though, so either it was total crap or I scared him away.