Many testers feel underrated; they don’t think they get the respect they deserve.
There are more reasons for this than suggested solutions.
One proposed solution is to define the profession more thoroughly, to get standards and certifications that can guarantee more than the bare minimum of test quality.
I am confident this isn’t the “good” path.

Main reason is that the role is so dynamic, it depends so much on the environment that standard processes or certifications often won’t be good practices. This is true for many professions, but some things are special for testing:

Expectations – there’s a huge difference between testing of a pacemaker, and testing of a personal blog. For some software the importance lies in reliability and security, and for others attractiveness is all that matters. Sometimes testing has traceability requirements, sometimes there just ain’t no time for planning.

Responsibility – even if the goals are clear, you have to figure out what is covered by other roles. If developers have good unit tests, you might not have to bother so much with regression testing.
There might, or might not, be usability, security or performance experts whose work you don’t want to overlap too much. Customer testing might cover requirement holes, and limiting contracts or “physical” access might set the scope. Regardless of your title you
might also deal with customer support, quality assurance or configuration management.

The solution for our status is small-scaled: do a darn good job by doing the testing that the product needs, and isn’t covered by others.
This will be valuable, and you will get respect, and higher status eventually.

This might mean creating automated regression tests, or very thorough testing of some details, or manual, lightweight testing beyond the requirements, or all of these and a lot outside and in between.
Your test team needs to figure out where and how you provide most value, other people can only guess (and help with expectations, responsibiities, goals.)

Long-term, we should promote testing so we get more talents, more diversity, more ideas; more, merrier and better.
University degrees might be nice, but our reputation will come with the products we help deliver.

“Wouldn’t it be cool if we could come up with a Quality Value for our products?”  said a colleague of mine. ”Yes! That would be super!” me and a couple of colleagues answered.

We had a lot of categories and data in our bug system; and perhaps most important was that we had a good data mining tool that enabled us to take the data and transform it by performing calculations and making aggregations of it.

We started out by analyzing the bug data and starting to come up with reasonable and weighted factors that would enable us to quantify the categories: Severity, Priority, Time to fix, Resolution, Bug Type,  etc. Then we constructed an algorithm that would go through all essential data and the result would be a numeric value, i.e. the Quality Value. We decided that the Quality Value should be in somewhere in the span 0-100 and scoring 100 would be the top Quality Value.
When we discovered anomalies in the result, we tuned the algorithm and the quantifiers so that the result would make more sense; and a lot of discussions were about the quantifiers and their impact on the result. After several iterations we started to get some reasonable numbers.

And by now you might have started wondering how we noticed the anomalies and why we could see that the numbers were reasonable? This happened because we already had a perceived value of the products so we were  biased by subjectivity (huh!).

Anyway, when we were satisfied with the numbers we had a marvelous Quality Value for each and all of our products! And I dare to say that we strongly believed in this Quality Value. Of course there was a constant debate on this subject, and we fought over how much certain categories should have impact on the overall value.
“My product” scored in the top interval so I was very pleased.
But pretty soon, somewhere inside of me, I heard a little voice whisper: “Ignoratio elenchi, ignoratio elenchi, …”

We were of course very naïve in that we believed that this metric would represent the quality of the product. Of course it didn’t!
A couple of observations:

• Bug data only deals with reported bugs
• Bugs are subjective
• Bug reporting is subjective
• Bug handling (management) is subjective
• Bug fixing is subjective
• All other quality criteria not caught in bug reports are not included in bug data
• Quality is value to many people that haven’t reported anything
• Bug data is only data about bugs (+ subjectivity)
• All of the above means that we really cannot compare bugs with each other

On the other hand, one conclusion we came up with that might be true was that the ability to care about the product and bugs were reflected in the Quality Value. And in some way, this meant that a high score indicated that the product was taken care of (see Broken window theory and quality). While this might have been true, we were ever so wrong with the idea on capturing the Product Quality in a single Value…

Read about Ignoratio elenchi

Also see The Quality Status Reporting Fallacy

The Black Swan, The Impact of the Highly Improbable by Nassim Nicholas Taleb

Eyafjallajukull started to spew out tons of ashes in the sky. Air Traffic over Europe was severely hindered and stopped totally for a number of days. This is a perfect example of a Black Swan Event.

What we call here a Black Swan is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable. I stop and summarize the triplet: rarity, extreme impact, and retrospective (though not prospective) predictability. A small number of Black Swans explains almost everything in our world, from the success of ideas and religions, to the dynamics of historical events, to elements of our own personal lives.

The author is a former professional trader who has been studying uncertainty, luck, risk and knowledge for many years. All models used in finance are built on the faulty assumption that it is possible to predict stock prices, oil prices and economy as a whole. We assume that we live in Mediocristan were all things are nicely described by Gaussian bell curves. While this is true for some distributions like height and weight of a human being, it is totally wrong for things like stock market movement, fortune of a person, development of technology and other things where it is often used. We need to understand that we live in Extremistan where our lives are largely dependent on random events and luck.

Taleb does not tell us how to predict Black Swans but he tries to help us make some of them grey and to anticipate that the future has many surprises for us. The purpose of the book is to help the interested reader understand why we should never believe in forecasters and stop trying to predict everything. The goal: avoid being a sucker.

Some of his statements are:

• The Ludic Fallacy: basing studies of chance on the narrow world of games and dice,
• Epistemic arrogance: our hubris for how much we know or are capable of.
• Narrative Fallacy: our need to fit a pattern to a series of connected  or disconnected facts
• Confirmation Error: looking for instances that fit your belief and find them

What can a tester learn from this information?

• Understand that estimations are very hard to make and using more advanced models like S-curves will not give us better estimations only more detailed. Understand also that it is impossible to know what critical problems will appear in what you are working on which implies that having an open mind when testing increases your chances of finding these unknown unknowns much greater.

• The beautiful word Serendipity – where you find things by chance – is behind most, if not all, human inventions. So using exploratory approaches in life as well as in testing will most likely yield a higher return than will following detailed procedures.

• Whatever models we create they will always be incomplete. So I wonder what it after agile and what is next after Exploratory testing.

• Like he says – I would rather be approximately right than precisely wrong which is why I like the low tech dashboard better than curves or tables of test cases run and failed.

There are many great insights in this book and I truly recommend it. After writing this article I seriously consider whether or not I have fallen for the Confirmation Error!

I stopped myself in the hallway just to ask a few quick questions:

Why do you like software testing?
- That’s not a short question! But it involves a lot of creativity, subjectivity, serendipity, critical and holistic thinking.

What’s your motivation for writing on a blog?
- I don’t know, I just can’t stop myself.

How come?
- Not sure, I think I came to a place where writing was the best way to learn things; and learning things is an absolute must.

What’s your favorite quote, right now?
- You can see a lot by just looking (Yogi Berra)

Any new paradoxes?
- No, just an old one: With check lists you do testing, with test scripts you do checking.

Who is the best tester?
- Can’t tell, and if I could, I wouldn’t. Testing is part of a team effort to develop software; it’s not a competition.

Which one of Kaner’s 101 coverages do you prefer?
- A broad version of #89: Potential Usage Coverage.

What bug would you like to find?
- A very important one that happens after very normal actions, but in a long, credible scenario. It should be on a non-released system, so it can be fixed before causing real problems.

What’s the recipe for Good Software Testing?
- 1 part preparations, 2 parts knowledge, 3 parts wide-open, trained senses, 4 parts hard work, and a big pinch each of creativity and subjectivity.

How many testers actually spend more than 5% of their time testing non-functionals?
- I think many have quality characteristics in the back of their head all the time; so for me it is 100% of the testing time.

Can you say something positive about ISTQB Certification?
- In the Advanced Syllabus there is an appendix with a lot of good recommendations regarding pitfalls in software testing.

What’s the most difficult virtue for a tester?
- Humility.

Can you think of any strange bugs?
- A crash bug when cancelling a dialog, but only if the dialog had been manually moved.

Nothing better than that?
- A crash when a value range dealt with 9.8922, 9.975, 10?

What’s your favorite accelerator?
- Windows+Break. Everybody doesn’t seem to know that this is the fastest way to bring up the System details.

If software testing is a social science, which qualitative methodology can we learn most from?
- Parts of Grounded Theory is a perfect fit.

What’s your favorite string to use when testing?
- “This is a tribute to the longest string in the world.”

What is the danger of solving problems?
- It wasn’t the important thing, and now you have stopped looking.

Do you have a lesson for software testing?
- No, but the last one in the book Lessons Learned in Software Testing is excellent, the one on Test Strategy Heuristics.

Do you want to be Deming or Fleming?
- Ah, you’re thinking serendipity-wise. Fleming and the penicillin, of course.

Do you have an alternative to SMART goals?
- I want my goals to be Vague, Ongoing, Motivating, Important and Trustworthy.

Where would you like to have an extra pair of eyes?
- I just have to steal this answer: I would give them to a blind.

How long does it take to become a really good tester?
- Don’t know. I might be there, and if so it happened gradually over many years. But it takes about one year to master sourdough baking, so it could be something similar for testing, if you have the basic skills, knowledge and motivation that are needed (don’t ask me about that!)

What’s your biggest weakness?
- Could be that I am easy to misunderstand. I often try to be serious and humuorous, at the same time; which can be very difficult to evaluate.

What development process are you using at your current assignment?
- Same as everybody else, a mix of Waterfall and Agile.

What do you have against Agile?
- Nothing really. It’s just so over-used, and doesn’t seem to mean much more than it’s something good (just like “professional”)

Don’t you agree that testing is a service?
- Can’t say it’s not true (and it’s extra important for testers to be service-minded), but I would talk about long-term team ownership several times before using service vocabulary.

If you were a young, aspiring software tester with a lot of spare time, what would you do?
- I would find an open-source tool I like (maybe a test tool?), and test it thoroughly, learning everything I can about the test approaches while using them.

Are you a certified tester?
- No, but I have a diploma from a basic software testing course.

What’s in your Software Testing Dystopia?
- Extreme specialization, same work all the time, no system overview, and cumbersome software with just some value.

What questions would you not want to be asked?
- What’s the most important aspect of software testing?

Anything else you would like to add?
- I am always on the testers’ side!

Thank you very much! Have a good day, and a nice vacation.

If you just do what’s on the list of things to do, I think you can accomplish decent things, but nothing great.
I don’t dare saying this is a general truth for everybody creating something new, so let’s focus on software development.
There are many management models, and many of them boils down to something like “do the things on this prioritized list”, and the list might be items in a detailed project plan, a Backlog, test cases or sticky notes.
As an attack, and a defense, of these systems I’m saying:

to accomplish something great you got to do a lot more than what is on the list

The reason for this is that you cannot in advance know about all important things and possibilities that are associated with what your team is trying to accomplish.
Great software has even more than the right capabilities, impeccable reliability, super-performance and perfect usability.
There are no Gods in software development that can specify the details in advance.
And if there was an almighty capable of doing this, it would take too much time to read and understand all items on the list.

If the input for the software is taken from the actual demands of the customers, you can give them what they want, but not necessarily what they need, and probably not something that will be very valuable for many customers. And the item on the list will probably focus on functionality, without all implicit details around quality characteristics that can sum up to a great product.

So the solution is to take height for this, for example by
* not performing (detailed) time estimation and planning
* allocate time for “creative slack”
* pad all estimates with an appropriate percentage
* add extra time for polishing (that must be spread out from start to finish)
* rotate free, unscheduled roles
* encourage everybody to pursue interesting things that emerge
* perform one unscheduled “good” thing for every other item on the list
* schedule vague items that can mean almost anything
* but most importantly: have skilled people doing their best to create valuable software

I believe that as a tester you do not value the same things as someone with a different role in the organisation. Each individual most certainly value different things, but I think there are some specific traits of the organisation that triggers a tester to stay or seek new employments.

As a test consultant you value different things. You move into different organisations knowing that you probably won’t stay long, still there are some areas that can be similar to what you value as a more permanent tester.

These are some of the things I value:

Test Approach: I value Exploratory test approach highly. If I enter an organisation with Script-based test approach I try to get it to move toward Exploratory.

Lovable product[s]: If I am to be an assest to the organisation I need to love the product or product family. At least have an interest in making the product something you take pride in.

Other testers: I think you work best as a team in testing. Therefore you co-workers become especially important. Do they have the same keen interest of testing as you? Do they also share the same passion?

Developers: I value the developers a lot. If you are working with good, open-minded, passionate developers it makes everything so much more fun.

Freedom: I value to have the freedom to do and say what I feel is right, even if it might hurt a bit for the organisation.

Trust: If I feel that my superiors and my co-workers trust in what I do and what I say, I will feel more confident and feel that my work brings value.

CEO: If the captain of the ship steers the vessel where I want to go, it makes a big difference for me as a tester. If the CEO think testers are important and that they add value, it is a key part for staying around.

Why did you seek an employment where you are today? When things are getting tough and it becomes hard to understand why you stay, what is keeping you?

What makes you stay?

It’s a truth that we can’t find all bugs, but is it really a truth that we can’t find all important bugs?
And it’s a cliche when used as answer to the (sincere) “why didn’t you find that bug?” question.
Testers are paid to find important information about what they are testing, and included are the big bugs, so you can fix them before the software is used in production.

Besides some good things about forthcoming test strategies, I hope you also can say: “if you have read the test plan and status reports it shouldn’t be a surprise bugs like this got away.” Well-informed risk taking is part of our business.
However, if there is a test that seems important, and would take a couple of minutes to perform, this argument is not valid. Testers (in a serious project) should try to perform all important tests that can be performed fast.

There are testing lessons to be learned if the answer is:
* we didn’t think about that scenario
* it wasn’t included in the requirements
* someone told us not to test that
* we had no idea that the customers where running that kind of configuration
* we found it, but didn’t report it well enough
* we didn’t put the bug there, and they hid it awfully well
* this type of issue can’t be found with automated tests

And there is the occasional invalid question, e.g. when the bug was reported in a good way, or when there was no testing time alotted for a late change.

Bottom line is: take the question seriously, and try to improve your testing to avoid this and other important issues (next time it will probably be another kind of issue…)

“Testing can’t be complete” might be the only statement all testers would agree upon.
This means that we only will run a few of all possible tests, and this is in many fields called sampling.
There isn’t too much said about qualitative sampling in software testing, so let’s look at what Grounded Theory says about theoretical sampling, and the need for adjustments based on reality:

“To say that one samples theoretically means that sampling, rather than being predetermined before beginning the research, evolves during the process. It is based on concepts that emerged from analysis and that appear to have relevance to the evolving theory.” Strauss/Corbin p.202

“The general rule when building theory is to gather data until each category is saturated.” Strauss/Corbin p.212

“The analyst might be, and often is, surprised at what he or she finds out when in the field. Variations that the analyst expected to find might not be there. New ones might emerge from quite unexpected sources. The analyst also must be prepared for this. But this serendipity is what makes doing qualitative research and analysis so much fun. There always is that element of discovery.” Strauss/Corbin p.233

It has been said that “a good tester is often lucky“, and half of it is not true; a good tester has the ability to see and smell important things, there is no luck involved. The other half is semi-true; a good tester make preparations and deviations to have bigger chances of luck, and this makes them stumble on important information more often (serindipity). A skilled tester thinks about many aspects at the same time, and sees things she wasn’t explicitly looking for.

But let’s follow the sampling thought a bit longer.
If we agree that sampling and serendipity is needed, does it make more sense that we should use multiple approaches, in order to come close to all the important information?
Does it make no sense to script all tests in advance and aim for 100% Pass?
If you agree that quality is multi-dimensional , do you want to spend at least some thought and time for all orthogonal aspects?
All due respect to equivalance partitioning, but would you feel comfortable just ’cause some part of the functionality has pretty good coverage?

Maybe we can get most areas and aspects partly covered by samples, and spread the samples in a good way to have a nice shot at the serendipity we need to find most of the important information.
Maybe the spread doesn’t need to be theoretically perfect, by relying on tester skill and serendipity, we might go for the fastest, and richest, tests, so we can execute many of them.
As in Grounded Thory (and exploratory testing), we can change the sampling strategy as we learn more about the product.

Exercise: For the next 10 bugs you find, note if the issue was something you expected to find, or if you were looking for something else.
Or do it as a team effort, but only count really important issues, and make it a game:
Prediction (manufacturing) vs. Serendipity (qualitative analysis)

There are many names for software testing strategies/activities/ approaches/processes; they can be risk-based, coverage-focused, exploratory, requirements-based, Super-TPI, TMM 5 et.al.
The names generally come from how the testing is performed or initiated, so I thought we should look at it from another angle, from the end of testing, from the results that we might know about a year later.
I like the content of Good Enough Testing, but the words are easily misunderstood; we don’t have definitions for Crappy or Decent Testing, and since that often isn’t enough it might be useful with Saturated Testing. (This is elaborated elsewhere with a different name??)

The term is borrowed from Grounded Theory (social science qualitative analysis), and means that further research in an area doesn’t give any important, new information, and therefore isn’t worth the effort.
The same can happen in ambitious software testing (if the test time isn’t radically shortened…): when important bugs no longer are found, and all important test areas have been covered, and no code changes are made, there is no need to continue testing.

I see three hypothetical levels of Testing Saturation:
1) more testing will not find information that would change the release decision
2) more testing will not find issues that later would be patched
3) more testing will not find bugs that we would like to have fixed

If I you use a mix of these as your goal, I have three warnings:
* Have you used all relevant information?
* Have you tried all relevant test approaches?
* Have enough people with different views been involved in (at least) generating test ideas?

To get a result you are satisfied with, you need to think about things like this early, you need a lot of skill and hard work, and a fair share of luck.

Well, to many people this is nothing new. But still, there are a lot of testers, and indeed test leads, that still think that Exploratory Testing is a technique that can be used in testing. To some extent, it has to do with that both Cem Kaner and James Bach have used this term amongst other techniques (e.g., in the BBST course material). But they have changed and updated presentations as much as possible over the last period of time.

According to Cem Kaner nowadays, the definition of exploratory testing is “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

And this is important. You can come a long way on reaching the style of Exploratory Testing just by treating testers as intelligent people; which is one of the most important factors in the definition above. In contrast to Exploratory Testing you have Scripted Testing that, in my opinion, treats testers as dumb people or even dumb machines. I think that this approach is devastating for our profession (even though I can somehow see the need for Scripted Testing in some places).

A technique is a recipe for solving a problem, whereas a style (or approach) is a way of thinking around a theme that stretches far beyond solving a particular problem.
So when we talk about selling in Exploratory Testing to managers or project stakeholders it is not a technique that we are selling; it is rather an acceptance of a mindset where testers are treated as professional and intellectual human beings that are able to perform Sapient Testing, and particularly in an Exploratory way. It is not about stakeholders investing in a technique, it is about them showing that they have as much trust in testers as they have in other intelligent co-workers of the project.