Initial thoughts on group testing Martin Jansson

I want to open for a discussion on pair testing or at least widen the concept. By saying pair, we say two people, but why should we limit us to that? Depending what your objective is a different number might be more applicable?

Several of the referred authors in this article have elaborated around pair testing as something that could be wider and be used in different ways, many of the things I have expressed below has also been brought up by them. Still, I’ve added some parts and changed the scenery a bit.

Looking back at pair testing

“Pair testing is a way of approaching a test design process by having two people test the
same thing at the same time and place, continuously exchanging ideas. Even without any
special method or process, the dynamics of pairing enables the generation of more and
different ideas than either tester is likely to produce on his own. It’s an effective
complement to individual testing.” a quote from a draft article by James Bach and Sam Guckenheimer [1] on Pair testing.

This draft goes through the basic concept of pair testing. They discuss how it can be used in different forms and where it might be useful and in which situation.

“Two testers and (typically) one machine.
• Typically (as in XP)
– Pairs work together voluntarily. One person might pair with several others during a day.
– A given testing task is the responsibility of one person, who recruits one or more partners (one at a time) to help out.
• We’ve seen stable pairs who’ve worked together for years.
• One tester strokes the keys (but the keyboard may pass back and forth in a session) while the other suggests ideas or tests, pays attention and takes notes, listens, asks questions, grabs reference material, etc.” a quote from an article on Exploratory pair testing by Cem Kaner and James Bach [2].

Here James Bach and Cem Kaner elaborate around exploratory pair testing, how they suggest the pairing should be done, benefits of doing it and some drawbacks and risks with it. They also identify that it can be done in different combinations and that it is suitable in different situations.

“Working alone is working without a safety net” a quote by Brian Marick in an article [3] on Pair testing.

He summarized what seems to be a discussion in a workshop, where several parties were present. Several examples are listed where pair testing would be good, but he also identifies areas when it is not suitable. Furthermore he points out the importance of it not being more than two, thus naming it pair testing.

Jonathan Kohl takes on a different approach in an article about Pair testing [4]. He gives some examples where he has worked with developers in pair testing sessions with great result. He provides some hands-on tips on how to go about this and things to consider. He is focused on working in pairs, but he also adds the concept of using this technique as a way to teach testing and to grow the relationship between different parties in the organisation.

“As in pair-programming, two people simultaneously test an application at the same computer. It can involve: testing an application from a customer’s perspective and more technical kinds of tests”, a quote from a presentation [5] about Pair Testing by Jonathan Kohl.

In this presentation Jonathan elaborates deeper around the usage of the Pair testing technique. He touches a range of areas and brings up some other aspects than previous writers.

One thing that I notice from the various writers is that you use pair testing or as a technique to accomplish different objectives. Each writer includes different things to the concept of pair testing. Based on that, I saw that a pair testing session could consist of different dimensions. One of these dimensions was the amount of testers involved in a session, which in turn made me question the name of pair testing. Instead I call it group testing, since I think it can be two or more based on the objective. Beside considering that there can be different amount of testers I saw that there are other dimensions worth exploring.

Dimensions of group testing

There are many dimensions that could be investigated and experimented with. Here are the initial dimensions I’ve identified and experience with:

  • Amount of participants – how many are part of the group testing session?
  • Role as a tester – which role do you play as a tester? Is it explorer/tester, documenter/observer, checker, coach/trainer/teacher, reviewer, manager or something else?
  • The scene – Are you running a specific user scenario or perhaps combining a set of user types? What scene have you painted that you are acting in? What personas are valid in this scene? What storytelling aspects have you considered when setting up this scene?
  • Mission of group test – What is your objective[s]? What do you wish to have achieved when the group test session is finished? Do you wish to find information quickly, teach, coach or train a member of the group, review testing skill, report information quickly, help trouble shooting or something else?
  • Note taking style – How are you taking notes? Do you take note in one single document that all of you write in simultaneously, is one of you taking notes only or are you all writing your own material?
  • Partner combo – Which other stakeholders are part of this group test session? Do you pair up with developers, project managers, line managers, CEO, sales and marketing, system managers and business analysts … who?
  • Lateral thinking style – Should part of the participants take on different lateral approaches such as thinking the opposite or wearing different thinking hats?
  • Personality mix – Consider how you mix personalities such as introverts, extroverts, slow thinkers and quick ones. Which combinations work better than others in your context?
  • Debriefing style – Do hold a debrief after each session or is it sufficient with the ongoing discussion you have?
  • Accountability focus – Do you have one driver who is accountable or do you have many driving at the same time?
  • Focus areas – Does each participant focus one specific characteristic (usability, security etc), a mix of focus areas between persons or possibly everyone focus on one characteristic?
  • Test Environment setup – Do you sit behind one system or test environment, one system or test environment per person or possibly a mix of systems per person?
  • Session duration – Do you do a longer session or short ones? Do change driver during or after each session, either way consider how this is setup  depending on the duration.

Conclusion

There are many dimensions to the group testing technique that need to be experimented with. My point to this is that I fail to see the point in limiting the technique to a specific setup, but instead let its content/dimensions change depending on context. There are a few major pros that you probably get by using this technique, no matter which dimension you alter. Some of these are:

  • an increase in creativity
  • an increase in cooperation in the team/organisation between different roles
  • an ongoing training of the participants in expressing themselves both in written and verbal form

Based on my own experience, I think this is an excellent way of testing with great result. Therefore I intend to dig further in this area to see what comes up.

References

[1] Pair testing (aka Extreme Testing) by James Bach and Sam Guckenheimer –  http://www.exampler.com/testing-com/test-patterns/patterns/XT-Pattern-jb.pdf
[2] Exploratory testing in pairs by James Bach and Cem Kaner – http://www.kaner.com/pdfs/exptest.pdf
[3] Pair testing by Brian Marick – http://www.exampler.com/testing-com/test-patterns/patterns/pair-testing.pdf
[4] Pair testing – How I brought a developer into the testlab by Jonathan Kohl – http://www.kohl.ca/articles/pairtesting.html
[5] Presentation on Pair testing by Jonathan Kohl – http://www.kohl.ca/presentations/pair_testing.pdf

3 Comments
Jonathan Kohl July 26th, 2012

Cool post. I’m glad you’re exploring this and I can’t wait to see where you take it.

I stopped using the term pair testing several years ago, and instead started using the term “collaborative software testing”. That was even the name of my blog for a while. What I found was that a pair was too limiting, because we had trios and even quads for certain testing activities, and it was incredibly powerful. In this: http://www.satisfice.com/articles/et-dynamics.pdf James and I discussed “pairing” for this reason, and he changed it to “collaborating”.

Here is an example: I was pairing with a test intern to track down an intermittent bug for a mobile app that had a complex server back end to support it. We were using a mobile device, a mobile emulator, a network traffic sniffer and we were tailing the logs on the app server. One would drive, and take notes to record the session, and the other would observe the app, and the other tools at the same time and point out issues. We would both express different ideas, or theories about what was going wrong and what we were observing, pick an idea, and test, then repeat when that idea dried up.

The lead programmer for the project came over to give us a hand, and stood behind us. He joined in, and then called a colleague over to do even more monitoring and debugging on the backend from his machine. At times, we were a trio or a quad, depending on whether the one dev was at his workstation looking into things from a different direction or not. We essentially had 4 people doing skilled exploratory testing with for about two hours. We tracked down the issue, using the combined brain power of the quad. We found the issue because of all of our thoughts, inputs and different skills. We eventually found it by following the thread that the test intern put forward, with the combined technical skills of me, and the two devs while the intern drove and took notes.

I doubt that any single, or pair of us would have found it. The test manager was astounded at the results, and said he had never seen such intense productivity. We were all absolutely exhausted afterwards, and we couldn’t collaborate with anyone for a few hours. I can’t imagine trying to do that all day.

In short, yes, “pair testing” is too limiting. Look at collaborating, and follow the dynamics of that type of approach to value creation for testing.

[...] Initial thoughts on group testing Written by: Martin Jansson [...]

Martin Jansson July 31st, 2012

Thanks for your comment.

Strangely enough I had looked at various elements of collaboration, but did not reflect on the term Collaborative Testing.

I called it Group Testing putting an emphasis on that it does not have to be just a pair, still I believe the emphasis should not be on numbers but on that it is a collaborative activity. So I like Collaborative Testing a lot better.