thoughts from the test eye

There are many good articles and reviews on how Session-based test management (SBTM) is used out in the field, see some of the references below. I’m going to try to analyse what I’ve experienced so far with Session-based test management. If you are new to these concepts please see reference [1] and [2] below to get started.

Learning & Education

When I try explain to others what is special about SBTM I emphasis learning and education. As a test trainer, test lead or an experienced tester it is a very powerful tool in teaching test design and communicating test ideas. The session report is one of the keys, but also the way you debrief. For instance, if you pair an experienced tester with a novice tester, you will be able to train both at the same time while at the same time getting a good result. You start out with a mission with a set of ideas that drive the group, you discuss what you intend to do and identify possible test ideas. As you start working your way towards the mission, you sometimes identify new things or change your original test scope. This collaboration teaches the novice how the expert works her way through the system, what heuristics that are applied and how new test ideas pop up. It also teaches the expert to better express and organise her thoughts. I’ve seen many times that the testing becomes a lot more structured when you need to express to someone, sitting next to you, what you are doing and why.

Speed

Since you are not bound to do just one test case in a specific amount of time, you will instead know that you have a certain amount of time to do testing on. A mission that you might follow if you think it adds value. If you find something else that is of higher value, you should investigate that instead. This gives the testers more freedom, but it also means that during a session (a time box) you are focused on testing that you think is valuable, no matter what. At the debriefing you need to explain why you didn’t follow the mission and what made you prioritize the other area. You will in a natural way try to focus on things that adds the most value, as you see it as a tester. This in combination with having prioritized missions will let you have a bigger chance in focusing on the right things. Comparing this way of working with how you sometimes do test cases, where you see problems in other areas as things that stop you from executing the “real” tests and where you report “no progress” when not being able to finish the test case. In those situations, you can sometimes be mentally blocked from doing what is expected of you. This might be an issue with the mindset of some testers and test leads, and not applicable to the whole test scripts vs. exploratory test discussion.

The indication that I see is that you will do a whole lot more tests using session-based testing than when you are focused on running specific test cases with predefined steps you should take. In some cases you might do duplicate tests that others also do, but a few extra pairs of eyes is most often good.

Accountability

Ad hoc and exploratory testing can sometimes be seen as unstructured and lacking accountability. Michael Bolton has discussed this in his blog in a good way here. One of the main ideas behind SBTM is that you instead of showing test cases that you have run, where you might or might not have followed the actual steps, is that you show in your session reports what was actually being tested and what was guiding you. The metrics produced is then based on what was covered and how.

Comparing SBTM with running a test case that is lacking the full repro steps, but where you give the freedom to the tester to test in the area. You use the full tool support so that you keep track on what revision, what environment and what context you think was interesting for a specific test. SBTM is lacking in the collection of metrics data based on that you are not able to state what areas each one of your tests in a session touched. Instead you enter what areas the whole session touched and might give an inaccurate truth. The error rate is still low since you have short sessions ranging from 30 minutes to a few hours, but the metrics on area coverage will be erroneous.

If it is important that you show what specific functions you have covered or what requirements you have covered, the actual Areas in SBTM might not be the too solution to this. Instead you should probably have tracability matrixes or something similar, as I see it. I’ve not explored this fully yet though.

Test areas

If you use SBTM, according to James and Jon Bach, you enter the areas that you are able to report time on in the coverage.ini file. It is good to spend some time in finding out the core areas and their hierarchical structure. The important thing is that you consider what you want to measure on and what is interesting in reporting, based on the metrics produced.

I think that it is good to setup the root areas as something along these lines:

feature | <area> | <sub area> | <sub area>

revision | <product> | <revision #>

environment | <hardware> | <hardware revision>

You will be able to add new areas later on, just keep track that you use the same kind of structure and that new areas are inspected every now and then. You can change the coverage.ini file without any severe issues other than having to change the dependencies afterwards.

Focus/Defocus Strategies

I’ve tried various setups in how to do the focus and defocus when running sessions. In one case I used a time slot for the whole group between a start time to a finish time where everyone where to round up and debrief. This strategy pushed the testers into delivering and being focused during this time. It also made some of the testers who had a harder time to start at a specific time to actually be part of these sessions. I used this strategy when in training and it worked well for that. If someone walked into the test lab to interupt it was easy to just tell them that we had training session and that they would be ready in about X minutes.

Another strategy was to ask each tester how many sessions he/she thought were doable during one week, then try to hold them to that commitment. This gives the freedom to the tester to run sessions at the hours that fits them. Still, it made it harder for controlled debriefs.

My recommendation is to try different strategies and see what fits your team best, perhaps even have a mix of them to fit each individual.

Satisfaction

Grumpyness is usually a common state of being, as a tester. I’ve noticed that the satisfaction level is different depending on what test technique you use (this is just a minor observation).

In one case we had a long list of test cases that were to be executed on a specific test environment. To be able to run these test cases you need to do certain setups and configurations that took quite some time. The test cases did not have anything to do with these preparations. The interesting thing was that most bugs, at that time, was in the area where we prepared to test. What happend was that at each stand-up meeting every tester complained about not getting anywhere and that they had no progress at all. The project managers thought we were doing a loozy job because we gained no progress. They thought we were slow.

So, instead of executing those test cases we ignored them all and ran test sessions in those areas where we explored what was causing these preparations to halt. A large number of bugs where reported and at the next stand-up meeting we instead said that we had made lots of progress in those areas where we had reported many Must bugs and that we were blocked from running those specific test cases. The satisfaction level was now high and we talked differently about things that stopped us, we found issues that stopped the release which was good progress.

Speaking for myself and some in my current team, I believe that you become more satisfied in as a tester when you use the session-based test management instead of running hardcore test scripts. You know that the important thing is that you test and focus on areas that that are important. So, hours spent testing or a number of sessions per week is something that you can talk about.

Test focused

The amount of time spent on testing vs. how much time you spend on planning is a lot different from the regular test scripts approach. This is the core of Exploratory testing and this is where SBTM shines. As a test lead you view your testers as intelligent enough to make their own decisions. The missions will guide them, but they will form their own decisions.

For instance, lets assume you have 10 weeks of testing. Do you spend a better half of that with planning what you should do and how to execute each test case? Or do you do just-in-time planning and setting up missions so that you always have a steering. Naturally I think the later is best. Overplanning is a curse for testing since we know that all plans change, requirements are just a whisper-game and testing is a social science. I’d rather spend time testing than discussing or planning what would be possible to test or what we cannot test.

Tools for SBTM

SBTM package by James and Jon Bach

James has a session.exe available from his web site. The documentation is good and you will be able to get started quickly there. Those who are fond of using simple text editors will like this approach. Included in the session.exe is various compilers written in perl. Paul Carvalho has done his own lessons learned here and updated the compiler scripts in Ruby.

I think the main idea behind this tool setup is that each user should own his/her processes and tools. So that you should make them fit your own company and way of working.

Test Explorer

This tool is recommended by James Bach. I test complex systems that only has a small part that has a GUI, this tool did not suite my way of working at all.

James Lyndsays tools

James L. has made an excelsheet that supports his way of working with SBTM. He has introduced a concept called Test Points that I find interesting. He has ideas on how to address regression testing and how to report progress. I recommend reading his paper [5] and trying out his tools.

Session Tester

This open source project [6] is run by Jonathan Kohl [7] and goes a somewhat different path than what Bach staked out from the start. Jonathan has been driving this project to steer in the direction that the tester community wants it.

I tried out the 0.21 release and noticed that lots of the ideas that I had gotten from using Bachs tools were implemented here, but with some changes or adaptions. I am not fond of using a text editor for all things, but want instead a powerful tool that is specialized to my needs as a session-based tester. There was no tool that fulfilled this requirement as I saw it. Therefore I offered my services to Jonathan Kohl to assist in the project.

We are now driving for a 0.3 release. Go to the website mentioned above to see what is included. If you want to be part of this notify me or Jonathan.

Courses

Rapid Software Testing

Some believed that this course would teach how to use this technique, but it does not. Instead it empowers you to become a great tester, which as I see it is even better.

SBTM Courses

I think there are individual consultants who offer courses in this, but it seems quite rare.

Documenting in a session

The skill of journal keeping is practised efficiently in SBTM. I see this skill as essential for a good tester and SBTM brings it out in a natural way.

I think each tester should have some personal freedom in how they document, but it is good to agree on some standard way of expressing yourselves. Bring up the discussion after a while that you have used SBTM and see if you can agree on something.

Organising missions and charters

There is a way of using the TODO lists in order to create template sessions, but I did not use this fully. Instead we had missions in a separate excelsheet together with other tasks. I figure missions are easy to include in regular Sprint logs or similar planning.

We let testers add new missions that they thought were necessary. Organising this in a good way will make it a lot easier to measure progress and to see trends. I’ve not explored fully where you could go with this, but I think you can use other tools such as Trac to do your planning in.

Summary

I’ve tried to cover a part of the areas where I thought previous reviewers had missed or not focused on. In order to make the full picture for yourself, you as well need to dig deeper.

I recommend exploring this way of testing. It will change the way you perceive testing and the way you plan. I think it is a lot more efficient that traditional test approach, but there are several holes to fill to make it even better.

If you want to own your tool by building on what James Bach created, if you want to buy a tool such as Test Explorer or if you want to participate in creating a tool for SBTM, such as Session Tester, is your choice.

References

[1] http://www.satisfice.com/sbtm/

[2] http://www.satisfice.com/articles/sbtm.pdf

[3] http://www.staqs.com/sbtm/

[4] http://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/

[5] http://www.workroom-productions.com/papers/AiSBTv1.2.pdf

[6] http://sessiontester.openqa.org/

[7] http://www.kohl.ca/blog/