Well, to many people this is nothing new. But still, there are a lot of testers, and indeed test leads, that still think that Exploratory Testing is a technique that can be used in testing. To some extent, it has to do with that both Cem Kaner and James Bach have used this term amongst other techniques (e.g., in the BBST course material). But they have changed and updated presentations as much as possible over the last period of time.

According to Cem Kaner nowadays, the definition of exploratory testing is “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

And this is important. You can come a long way on reaching the style of Exploratory Testing just by treating testers as intelligent people; which is one of the most important factors in the definition above. In contrast to Exploratory Testing you have Scripted Testing that, in my opinion, treats testers as dumb people or even dumb machines. I think that this approach is devastating for our profession (even though I can somehow see the need for Scripted Testing in some places).

A technique is a recipe for solving a problem, whereas a style (or approach) is a way of thinking around a theme that stretches far beyond solving a particular problem.
So when we talk about selling in Exploratory Testing to managers or project stakeholders it is not a technique that we are selling; it is rather an acceptance of a mindset where testers are treated as professional and intellectual human beings that are able to perform Sapient Testing, and particularly in an Exploratory way. It is not about stakeholders investing in a technique, it is about them showing that they have as much trust in testers as they have in other intelligent co-workers of the project.

Let us assume you are a test lead and you have a group of testers. Some are totally new to the profession and some are old and experienced.

In the scripted test environment you might setup a test matrix, plan test cases and allocate them among the testers. Some of the testers might have been involved in creating test cases. Some might create test cases along the way as they do testing, but in the extreme scripted test environment you plan the tests before you get the build. Those that were not involved in creating test cases will just execute the tests.

Where is the learning process in this? Is it limited to just a few of the testers or is it in fact the whole group?

If you introduced pair-testing combined with the scripted testing there might be some collaboration that would stimulate learning, still if you are to follow the script there is no room for going outside the path.

When we do session-based testing you might identify risks and outline which missions to do. Testers will during the test session practise their way of expressing what they test, how they do it and what they find. Then during debriefing there is a natural way of giving feedback. Overtime you can look back on what you did and how you have evolved.

If I look at a group of testers who have been running test sessions, I see enthusiasm and sometimes passion keeping them from stop testing. Compared to those who have been executing predefined test cases, I see lack of will and boredom. The learning will be affected by this.

I have seen a few times that when giving out missions for test sessions to a group of testers who were inexperienced at exploratory testing, but instead were very familiar with how test scripts where run, there was a lack of courage in going to unchartered territory. I heard objections such as “But we have no test cases for this area, we do not know how this works.”. Can it be so that exploratory testing also brings courage and decrease fear of learning new things?

I have written tons of test cases over the years and I would say that you do learn by designing a test and preparing it to be run by yourself or by someone else. Still, it is not the same kind of learning and not the same kind of satisfaction (at least for me) as when writing on your charter.

What if scripted testing began making charters the same way as an exploratory tester do in session-based testing. Would they also increase their learning curve? What other changes would we see in the testers behavior? If the script was the mission and going outside was opportunity, would we infact see even better testing that what we see today from exploratory testing without the script? Is one of the keys the charter/opportunity, thus the fact that it is ok not to do the script?

When looking for personel in general it is common that we want passionate people who love their work. Most passionate testers that I read about are usually part of the context-driven movement. Can there be testers out there that are really passionate about how they work in the heavy scripted test environment, where someone else does the thinking, writes the script and then hands it over for execution? What personal goals do you have when doing these tasks? Is your main focus in moving out of that zone so you also can do the thinking, write the test script and pass it on?

In the ET heavy environment I tend to see fewer roles and career paths, is this true? There are test leads, test managers and testers. Everyone want to test since that is their passion, no? You have fewer administrators and more doers.

In the scripted test environment I see more different roles and career paths. There are experts in certain domains, writing test cases. There are many test leads who spend their time trying to lead what is to be tested and exactly how. There are test managers allocating the very few testers in executing these tests. There are more administrators and less doers. A few times I’ve seen almost 90% administrators and 10% testers. With administrator I mean someone who has little focus on actually doing any tests themselves, and in some cases do not want to. One common management idea is that anyone can assist with executing a test script. This means that anyone can be a testers, thus you are not special in being a tester and you are easily replaced. There is no skill in being a tester since you just follow that script, no?

Are you then passionate about being a script tester?

In many organisations it is hard to change how you are working. You might be bound to certain CM tools, how things are expected to be planned, documentation systems, management expectations, project management expectations and so on. In many of these traditional environments you might also use the regular test plans, test matrices, test specifications, test cases with expected result and test record.

If you are doing scripted testing it might sometimes be hard to handle the cases that fall a bit outside the scope, that mess up the planned activities. Some might say that it is better to stick to the plan and leave the vague and hard to reproduce thing for later.

To totally change the way you work might affect many things, so you might want to start out with small changes. If you are in such an environment and consider how it would be possible to try out exploratory testing as an approach to testing I have a few suggestions.

One way is using your current test cases as a guide to where you intend to test, then you use exploratory testing on those areas. The test cases will in this case just map lots of areas that need to be looked at and what you actually cover and what you expect is up to you. This means you will certainly skip a lot of the content of the test case. If this is accepted that is perfect, if not see how you can get away with it.

Another way is executing the test cases and each time you find something outside the test case or something fishy along the way, you create a work page/task for this issue. You then assign someone or a small group to dig deeper into this area using the exploratory test approach. All issues that are outside the regular plan are handled as an exploratory test.

You will report progress and result as usual, noone will know that you tried a new method in secret.

Each time you use exploratory testing you do it as a defined session, you can google how this is done. There are a multitude of good articles out there on how this can be done.

Exploratory Testing in its purest form is an approach that focus on learning, evolution and freedom.
Cem Kaner’s definition is to the point: “Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

ET in real life is a collection of ways of testing, sometimes used as example implementations of the approach, sometimes used for testing that don’t follow a script, and sometimes as a synonym to ad hoc testing (because that word has become increasingly under-rated.)

Scripted testing in its purest form is an approach that focus on precision and control. It is yet to be defined by proponents, but a benevolent try could be:
“With a scripted testing approach the testing effort is controlled by designing and reviewing all test scripts in advance.
In this way the right tests are executed, they are well documented, progress towards 100% execution can be controlled, and it is easy to repeat tests when necessary.
The scripted approach is not dependent on many tester heroes, and can take advantage of many types of resources for test execution, since the intelligence in the test scripts are created by test design experts.

Scripted testing in real life mostly means designing test scripts early, and executing them later, and these scripts have quite detailed steps, and a clear expected result.
The terminology is rich, complex and sometimes confusing, since they at least can mean approach, style, method, activity and technique; and these are in reality so connected and intertwined that distinctions aren’t necessary or helpful.

So are the distinctions important?
I think they can be, especially if the words are used without details, e.g. in statements like “Exploratory Testing is the opposite of Scripted Testing” or “combining exploratory and scripted testing”.
Both statements can be true, because the first talks about the approach, and the other about methods.
By understanding the different meanings of the words it is possible to get a more nuanced debate, and to see other combinations, e.g. test scripts with an exploratory approach, or scripted approaches with elements of ad hoc testing.

The intention of the used method shows which approach you are using (inferred from Cem Kaner’s Value of Checklists… p.94):
If test scripts are used to control the testing, it is a scripted testing approach.
If test scripts are used as a baseline for further testing, it is an exploratory testing approach.

It would be nice to have a solution to this semantic mess, but I don’t think it is feasible to always attach approach or method to Exploratory Testing and Scripted Testing (or to distinguish between upper case Exploratory and lower case exploratory.)
It is extremely difficult to give life to new words, but I do have some hope in the clarifications by testing vs. checking, and less hope for a renaissance of ad hoc testing.
A start would be if more people are aware of the different meanings, and are more precise when necessary, and eventually the problem will dissolve, in 25 years from now.

From a managerial perspective without knowing too much about testing, your sole experience comes from the scripted test environment…

What does Scripted Testing include?

• Control over what is to be tested, in the sense that you have a clear coverage of test cases on certain areas. Reports where you can see exactly how many test cases are left to do before you are finished.
• Hierarchy among testers, with different roles playing their part in the system. An excellent way to have career moves. Test leads who do the thinking and the others to execute.
• Scalability in personel by easily bringing in new people who can execute the test scripts that the test leads have written. Getting new resources in the middle of a project is easy since you can take almost anyone.
• Test Management Software. If you want your testers to have the best tools, there are tools that cost several millions. They will generate excellent reports, they can automate so that the boring repeatable tests do not have to be rerun manually.
• Education. There are lots of certification to be had and many firms who can teach you the essentials.

What does Exploratory Testing (ET) include?

• Flatness in organisation. The testers make tests as they go along. They do not need the traditional test leads. It is uncertain what kind of career exits you have with this kind of focus.
• Chaos. You have no idea on how you are going to test. You have not planned everything out in detail before you start testing. You cannot report exactly how long time you need and as much time as possible is not good enough.
• No scaleability. You only want testers. Not anyone can be a testers. It is hard to just get anyone to help out since you cannot use just anyone from the organisation, they must know the basic skills to help out.
• No real education. You cannot get certified. None of the major consultants teach this. There are courses, but they are new and few teach it.
• No real Test Management Software. The major software vendors have no tools for this. The software is seldome made by commercial vendors.

What do your managers think about this? What sales pitch do ET have for management? I’ve twisted what ET can include to bring out the worst of it, but I think that an inexperience test manager with his roots in the scripted test environment has at least part of this view. Cem Kaner has a good presentation called A Tutorial in Exploratory Testing. That presentation do sell ET really well. If you are going to try to get your organisation to start using ET and intend to do some propaganda for your test manager Kaners presentation is a good start.

Sometimes it is said that each test case must have an expected result, or even worse, that each step of a test case must have an expected result.
This is the extreme of scripted testing that I dislike for two reasons:
* It takes a lot of time to write and follow detailed test cases; time that is better spent testing things you haven’t had the time or enough information to thoroughly document.
* It takes away the freedom of the tester to try things a bit different, so we will learn less, narrow our thinking, have less chance of finding new, interesting information, and it will be boring to test.

I understand that the reason for the details is that you want to be sure exactly what has been tested.
But the problem with this reasoning is that in software testing, there are so many things that aren’t tested. You are not testing on the exact same data that your customers use, you are not testing on the exact same hardware and interacting software, and you are not testing in the exact same way as the customers will use your software.
So we can be sure which tests have been run, but that doesn’t necesarily mean it will work for the customers.

This is an example where (Bad) Process is way too much more important than Content.

I prefer a lot of one-liner test ideas, that either can be used in vague test cases or as starting point for Exploratory testing.