Preparation

The hardware setup of the testlab was 1 server and 4 laptops brought by Compare TestLab, 5 laptops by Adecco IT Konsult and finally 2 laptops brought by James Lyndsay. Many participants brought their own laptops. Before the conference I worked some on setting up a wiki, bug system etc. I tried to find a balance in not over-planning, but noticed later on that I had written down too little information making it hard for the participants in getting started and in knowing the overall missions.

By having a base of 11 laptops we were able to bring at least a few testers into the lab. Many thanks to Compare TestLab for sponsoring with both hardware and someone assisting us with the testlab environment, namely Torbjörn Wiger. Big thanks to Adecco IT Konsult for letting me utilize their laptops in the final minutes of preparation. Big thanks to James Lyndsay for being part of this confused setup and being patient with acting in the unknown and unplanned.

We had selected three applications as part of this setup, namely Sikuli, CiviCRM and Red Notebook. They were chosen because they were fairly buggy, open source and would make a good subject for interesting testing.

The planned agenda for the testlab had been published here with a bit more detail on what I had in mind here. During the conference the agenda changed and its content became something different, still within the planned concepts.

When we arrived to Let’s Test we started to setup things up almost immediately. I got help from Steve Öberg with some installation and by Torbjörn Wiger to set the server up. Doing everything on your own is hard so the help was greatly appreciated. I did not feel that we were fully ready when we stopped on Sunday night, but are you ever fully ready?

Day 1

During the day I talked to more speakers and participants to be part of the events in the test lab. I was very dodgy in details, since I only had rough ideas on what would happen. At 20:00 the door open to the test lab. Quickly afterwards Rikard Edgren, Simon Morley and Christin Wiedemann arrived and agreed to take a team and an application each as their focus area. The first event was collaboration test planning using Corkboard.me to plan in.

Participants started to arrive and we assigned them to the teams. Each of the team leaders did things differently based on what the team members needed. I noticed that some created missions/charters to be tested later while some wrote down their sessions directly. The content of each corkboard.me was different for each team depending on application/system and the team members. At the end of the collaboration session we held a debriefing where each team talked about what they had learned and experienced. Rikard, Simon and Christin held the groups together in an excellent way.

Start of the next session “Group Test Experiment”.

I held a short presentation about the various dimensions of group testing. I selected not to direct them in this, but left them to choose themselves what they wanted to try out. After a while I interrupted the teams and unfolded a scene/scenario that was about to happen. It went something like this “I, the test lead, am going to the release meeting for our products in 15 minutes. I want you all to give me the three most important bugs, where each team focuses on their application”. After roughly 15 minutes, I asked each of the teams to give me a 30 second report. Being able to give the essence of what is important is good to practice and often. Each team delivered some important facts all with a different pitch and undertone. The time was roughly 23:00 and it was time for a last beer before bedtime, at least for me.

Day 2

Discussion about having coaching session with Ilari Henrik Aegerter and Anne-Marie Charrett. Talked to more participants to get them into the lab. I let Rikard test some of the concepts for the evening event in the lab, and got some good feedback which made me change the agenda somewhat. Then I started setting up the lab so that it was possible for more teams instead of the initial idea of just two big teams, which must have been a good decision consider we were at the most close to 60 people if I read the tweets right. I selected Sikuli as the application/system for the evening event as the only system under test. A few people started to arrive at 20:00, but it was not that many to start with… so I considered if I had made the right move. All of a sudden the tables started to fill and the test lab was almost full.

I did a short presentation about the focus for the evening event, at least what I thought at that time. The idea was to have coaching of the teams in the first hour and later on to try out some group testing experiments. But I had to change that since the expectations of the participants were so diverse. Instead I said we would do a little bit of both during the whole evening, but with the major mission of creating a compelling status report.

Even more people arrived, who we filled up in the existing teams and by now we had 8 teams of 4-7 members (if I remember correctly). The context of the test lab had changed a bit from what my initial thoughts, I knew that the plan never survives when reality hits it. We had not put that much focus in describing in detail the full content of the test lab and how to get started in a good way. We thought that it would be easy to fill in that information as we went along, but that never became true.

Ilari and Anne-Marie came up with an idea for how they could coach the teams, which we tried almost directly. I told the teams that “HR (Human Resources) had decided that each team need to appoint someone to be tested.”. This act caused some confusion. Just prior to that I had told the teams that the focus was on creating a status report. So we had conflicting missions. Ilari and Anne-Marie did an excellent job even though it was a tough situation. Some teams were very focused on the testing mission and did not feel they had time for coaching, while others took it in and used the coaching to affect how the testing was done in the team.

Ben Kelly had joined in the test lab and I explained the current context and I explained roughly what I wanted him to do. I general I gave very thin outliers in how they could act in the test lab scene. Instead letting each player expand what was possible. Ben talked to the teams and coached them about what our mission was and about the creating of the status report.

During all this me and James were interacting with the teams. James gave some teams missions while I sometimes instead tried to confuse them. All-in-all we tried to simulate a situation where confusion and the unknown is our arena, but where me and James were roughly sure of what the other was doing in order to act on it.

After roughly 30 minutes of coaching by Ilari and Anne-Marie, I addressed the teams in the role of a project manager stating that “We don’t have time for coaching! I want those status reports!” The confusion was somewhat diminished and a renewed focus was visible. We decided that we would round up and hold debrief in about 40 minutes time, which was at 22:10. Some teams worked in X-Mind, another in the Corkboard.me UI, another in Post-It notes and yet another in Notepad. Each team focused on different areas, but some had found the same obvious issues. Some had found many issues and some found less.

It was my impression that Sikuli was quite buggy and that it really was not able to solve the automation in a good way without many workarounds. Many in the teams did not trust the application.

Even if we tried to keep the teams confused, that the team members where new to each other and new to the applications as well as to the test lab environment, they managed to produce quite good results. The traditional role of a test lead/manager could be questioned based on this or at least investigated further.

A few times when I confronted the teams and asked for specific results or information, where time was a factor, I noticed that some of the seemingly experienced context-driven testers returned questions, poking for more information. This is most often a good trait, but when time and timing is important you need to consider which questions you really want to ask that you believe bring value. Is this a common mistake that context-driven testers do? Still, a lot of the traps and confusion that was thrown on the teams was deflected in an excellent way.

Ben moderated the debriefing by asking for clarification then asked me in my role as a project manager if I was satisfied with the result of each time. It was a difficult question. From a test lab facilitator point of view, I wanted the teams to have a good time but also learn new things. I believe that objective was met. As a project manager I would probably had wanted to see a more consistent reporting style even though the information was on different areas. I was a bit split in what I actually thought here.

By setting up a scene it was easier to let the teams and their team members manoever in a changing context. It was very hard to balance between giving detailed instructions and giving a general outline in which they could act. Some teams embraced the scene quickly, while others found it harder. I believe it is important to be able to roleplay like this as a tester, to be able to act in a scene painted/described by someone else. It lets you explore so much more, with different mindsets.

During the Keynote of Julian Harty, he refered to “the testlab” but at this conference it was “a testlab” namely The Let’s TestLab, which was focused on learning, collaboration and experimentation. I am not sure we were fully doing open source testing nor any scientific method, is that still ok? I think we should embrace that many other testers run testlabs the way they want it to be. Otherwise we limit ourselves in what we can learn.

On previous test labs, I believe, the emphasis has partly been on the systems and applications. On this test lab it has been on the testers themselves and their collaboration. They have been able to explore some dimensions of group testing, note-taking techniques, collaborative test planning/preparation and finally seen different styles of status reporting. As a tester we know things could be different, but also that there is no one true way of performing a task. Hopefully this year Let’s TestLab showed that.

A big thanks to those who wanted to be part of the scene and acted in it. Another thanks to the speakers who helped facilitate with me and James. Finally, thanks everyone who participated that made it great!

On 7-9 May the Let’s Test Conference will take place. During the day there will be lots of interesting tutorials, keynotes and sessions. During the evening the events will continue. One of these activities is the Testlab, that we call Let’s TestLab. Initially I misunderstood Henrik Emilsson when we started to organise the lab. I thought the evening event was the testlab. At the time I did not consider anything wrong with having 150 people or more in the testlab. As I saw the evening event program I considered how could I compete with such a fine setup of activities. Well, this is a conference with many context-driven people. We will have different interests, expectations and focus area. So instead of admiting defeat, I considered what elements that we could add to the testlab to bring a great crowd, but perhaps not all of them.

Here is the line up of events in the testlab:

• Collaborative test planning
• Group Test Experiment
• Test Competition
• Making a compelling status report

Collaborative test planning

We will create possible charters, missions for the coming testing in the lab so that those who wish can practise different testing techniques. Everyone is invited to share his or her idea on how to plan testing in through collaboration.

Part of the line up in this event is:

• Rikard Edgren
• Christin Wiedemann

Group Test Experiment

The context of the testlab will be single testers or groups of testers going in and out of the testlab in different time intervals. Each tester will be unique in the sense that they bring different level of experience, skills and approaches to testing. Based on this, we will start experimenting with group testing. I think we have limited ourselves too long with the setup of pair testing. Going back to the early recommendations and experiences from Brian Marick, James Bach, Cem Kaner, Jonathan Kohl and many others, the setup is nearly the same. With new tools and techniques appearing over the years, some assumptions could be questioned.

Let us assume that there are different aspects and combinations of group testing that serves different purposes, we can say that there are different dimensions that could be explored.

Here are a few dimensions that we will experimented with:

• How many testers (2 or more)
• What role you play as a tester
• User types, User Scenarios or storytelling
• Mission of group test
• Note taking techniques
• Partner combination
• Lateral thinking aspects
• Personality types
• Debriefing techniques
• Accountability
• Focus areas or Characteristic focus
• Test Environment
• Basic Configuration Matrix
• … and new ones that we find along the way

We will explore and experiment with different tools that we use when group testing and share experience on what works in what context. We also experiment with a few pre-defined group test setups such as:

• Cinema testing
• 15-min test run
• Coaching a group of testers
• Wolf pack concept
• Testing Dojo

Part of the line up in the test lab is:

• Ann-Marie Charrett
• Markus Gärtner and Meke Mertsch
• Johan Åtting

Test Competition

Can you really compete in testing? Can you compete between two or more teams?  Can you really estimate the value of one piece of information against another? Well, it depends.

I have been an Ultimate Frisbee player for 34 years. I’ve not played for some time, but once a frisbee fan, always a fan. I think the same goes for testing. There are many things that I feel is similar. Craftmanship/sportmanship and passion is major part. Here is one description I like:

Ultimate has traditionally relied upon a spirit of sportsmanship which places the responsibility for fair play on the player. Highly competitive play is encouraged, but never at the expense of the bond of mutual respect between players, adherence to the agreed upon rules of the game, or the basic joy of play. Protection of these vital elements serves to eliminate adverse conduct from the Ultimate field. Such actions as taunting of opposing players, dangerous aggression, intentional fouling, or other “win-at-all-costs” behavior are contrary to the spirit of the game and must be avoided by all players.

One important element in Ultimate Frisbee is the spirit of the game, which you can see more in detail here [1]. Passion and humility as a tester are important traits, the spirit of the game concept might help us here.

So, my idea for a tester competition will be based on some of these ideas. Two teams compete against each other in form of best bugs and session notes. The two teams go through the opposite teams material and conclude who they think should be the winner with a good reason why.

Make a compelling status report

As the last event in the testlab we want to investigate how we can make a compelling status report for our stakeholders. Having many different testers, session notes spread all over, half-finished bug reports and test ideas half-finished… can we create something that is still valuable to someone? I guess this is a common situation at any lab at any company, still we will dig deep into how to go about this.

Part of the line up in the test lab is:

•  Ben Kelly

References

[1] Ultimate Frisbee Rules –  http://www.cs.rochester.edu/u/ferguson/ultimate/ultimate-simple.html

Many of us has a title that is connected to what we do at work. Every now and then I come across titles that makes me wonder what it really means. This time it is one that has been around for some time now: Thought Lead, what does this mean? I would not be suprised if there is a good explanation of how this title first came into place, but I do not think this is well known and I have not researched it.

Sometimes when I read status reports or hear project managers talk about testing, I hear that “testing is blocked”. What do they mean by that? When I delve deeper in what they are talking about I sometimes see that the progress on workpackages for the testing team or testers in a team have been combined with the result of a test case such as pass/fail/blocked. But that is hardly the truth? As a tester we do a lot of things, not just running test cases (if we even do that form). If we say that one test might be blocked in the sense that it does not provide value in testing further there. That does not really mean we are blocked? Is a blocked test when you cannot get the answer to a certain question you have about a system? Does it mean that a specific test could not be done, but perhaps a similar one be done that would provide as much value?

If everything around you is blocked and there is nothing you can do as a tester, then you can perhaps say that “testing is blocked”. But what about preparing for coming tests? What about automating some of the things that you have found? Did we then mean that we were blocked in the sense that we did not come up with any ideas on things we could do? Well, perhaps you can look at various test idea triggers that can get you going?

Progress in performed test related workpackages and the result of testing are two different things. Combining them does not really give a good picture of what is happening. So, before stating that “testing is blocked” consider if you actually mean that you have no progress or that you are unable to test in a certain area. Based on what you inform and how you communicate, stakeholders will make decisions based on that.

I usually tell my stakeholders that they should not worry about me or my test team being blocked in progress, there is usually something valuable to do. Still, if I say that some tests cannot be performed or that they are blocked, where the result of those tests are important they need to assist with removing those obstacles.

During a daily stand-up meeting in a test team I was part of, nearly all testers reported that they were blocked and that they had no progress in their test cases. We had a long list scripted tests that needed to be executed according to the test project manager. I investigated what they actually meant when they said that. Interestingly, most of them had a test cases that wanted them to go in a certain direction through the system. What was blocking them was bugs. They found lots of bugs, but did not feel they had time to report them because they really needed to gain progress in those intended test cases. At that time we had to work overtime if we did not have enough progress in our test cases. I asked them to instead of trying to run the test cases they use the test case as a test mission or charter to guide them where to go. When they saw fire or smoke, they followed that trail and documented (session notes) what they saw and what happened, but also log all bugs they found. From then on not one reported being blocked in the daily stand-up meetings. The flow of reported bugs increased and the progresss was fine (in the sense that test cases were executed as far as project management thought). Stating that you are blocked might in fact be how you work that does not enable freedom and creativity.

If we were to measure something as a manager of testing, would we then see a decline in testers reporting being blocked when using methods such as SBTM instead of scripted tests that report pass/fail/blocked?

I see humility as a very good virtue.
It is something I have failed miserably at, partly because it is easy to think something is bad just because there are many problems.
I think it’s a common fallacy for many ambitious testers – you are last in line, maybe with lower status, you want to get heard and become too persuasive…

Humility tend to give better collaboration, and thereby better results. A humble tester more often changes her mind, which is necessary.
The complexity of reality demands a humble tester.

Here is my list of humbling experiences; things I think have made me a better tester.

* write requirements
* programming something bigger than an exercise
* manage projects
* customer report of an important bug you should have caught
* experiencing too little time to test the very important stuff
* failing to explain why your testing is good
* reading something you wrote some time ago, and realizing it’s very shallow
* understanding your lack of humility
* realizing you were wrong

Which ones do I have left?

I spent this weekend in nice, dark, foggy Ringsjöstrand for the third Swedish Workshop on Exploratory Testing. Johan Jonasson, Ola Hyltén, Anders Claesson, Oscar Cosmo, Petter Mattsson, Rikard Edgren, Henrik Andersson, Robert Bergqvist, Maria Kedemo, Sigge Birgisson, Simon Morley.

The format is LAWST-style, which means a presentation is followed by a facilitated discussion, that goes on as long as it brings value.
The theme for this event was Teaching Testing, and the abstracts can be downloaded here.

I did the first presentation using my current assignment as teacher together with Henrik Emilsson on a 2-year higher vocational study for testers.
We have a lot of exercises to make the knowledge stick, and we primarily use open source applications to make it more real.
I shared how we find suitable applications and tie them to something we are teaching, and I showed some examples of applications we have used.
The discussion that followed lasted a couple of hours and spanned many areas: course literature, complexity of software, unreal situations, authorities, ISTQB, tools, how to teach so they learn etc.
I learn a lot by teaching, and also by discussing the teaching.

The second experience report came from Johan Jonasson, who is involved as instructor in BBST online courses.
The videos are available for free, but the best thing with the education is the realistic problem-based assignments and the peer reviews.
Discussions followed for a couple of hours and included cultural and online difficulties/opportunities, how to give feedback in writing, and a summary of the slides material:
“a condensed library of test information that will broaden your views.”

After a break, we had six eight-minute Lightning Talks, where you can’t go deep, but a lot of ideas are brought up:
Anders Claesson shared his model for making sure students really learn what is thought.
Sigge Birgisson talked about his efforts to create a quality vision/model/goal for deliverables.
Rikard Edgren wondered why no one (except Oscar Cosmo) test charisma when we all know it is important. (Emilsson came up with the name Charisma.)
Henrik Andersson showed that objectivity is an illusion, so we should focus on the inherent subjectivity in test selection, execution, interpretation.
Robert Bergqvist wondered if Exploratory Testing has been more accepted, or if it just is a word in fashion.
Simon Morley has a great idea on a Groopman-inspired book “How Testers Think”, although he denies he will write it.
Then we had dinner, bubble pool, a lot of conversations on many subjects (“lean is waste for testing”), perhaps a small beer, and a good night’s sleep.

The second day only had room for one topic, which was Simon Morley on “Mindset Changes: Changing the direction of the oil tanker”, about how to teach the “right” view on testing to both managers and testers.
Many suffer from test case counting, and they have to be cured in different ways.
Simon says to testers he don’t want to hear any numbers, and thereby force them to talk about what they have seen, and what they haven’t done.
He does presentations to managers about problems with number focus, and after that he can use dashboard-like reporting.
The most important dimension might be trust.
Consensus in the room was that gut feelings are better for making decisions, but that we don’t have the proper words to talk about this.
A lot of examples and ideas were shared, but we didn’t reach any solutions before the time was up…
“if communication isn’t good, it doesn’t matter what you do”

Eleven happy check-outs with a richer network and knowledge; SWET has charisma.

I spent two days in Malmö attending a developer conference with a fantastic test track (put together by Sigge Birgisson.)

I did a presentation on Curing Our Binary Disease (slides, abstract), which was much better received than I hoped for (I thought it was a binary love/hate talk)
Good questions and talk about being inside the potato, how do you know where you are?

Pradeep and Rikard in front of Øredev projector

Pradeep Soundararajan talked convincingly about caring for the users, e.g. by using Twitter as a source for test ideas.
He also said the context-driven community has moved from Pass/Fail to “Is there a problem here?” (still a binary question though…)

Shmuel Gershon shared an experience report of a 100% exploratory testing project, with qualitative reporting and everything (“it’s called quality report, not quantity report”.)

Gojko Adzic talked about testers needing to adapt to how development is done now, and put most of their time ensuring others write good automated tests.
“There’s so much mistrust in the processes.”

Henrik Andersson wants, with right, a diversified testing team, and therefore only hires context-driven testers
Just joking, he said people can read the same excellent books, as long as they think critically and learn other things.

Selena Delesie explained how to focus the testing effort on customer needs; “testers are information radiators”.

Zeger van Hese made the presentation that I enjoyed the most. A thoughtful walkthrough similarities between the fine arts and testing.
“The Hungry Eye – thoughtfully looking at software”

Janet Gregory highlighted five enlightened areas since Agile Testing was released: feature acceptance, collaborative automation, large organizations, distributed teams, continuous learning.

Outside the venue I had many interesting conversations, there were testing games, food and drinks, Black Viper and even imitations, and all in all a very nice experience.
Thumbs up for a testing session with Pradeep and Shmuel that showed (the need for) different note taking styles.

The presentations will be made available online at www.oredev.org/

I know of some testers who are pointed out by others to be Bug Magnets; people recognized for their ability to somehow draw bugs to them. Bug Magnets can be found in many workplaces and I bet that you know of someone that falls under this description. I have been appointed a Bug Magnet by some and it have made me thinking on what this phenomena boils down to.
Is it luck? Is it faith? Is it an ability that some are born with and some aren’t? Can you learn this ability? Can you improve it?
I have wondered about this for some time.

However, this summer I had a revelation when I watched an episode of the marvelous TV series “Homicide: Life on the Streets”:

Homicide: Life on the streets
Season 1 Episode 9 “Night of the dead living”
Det. Frank Pembleton and Det. Tim Bayliss

Bayliss [sits pensively. To Frank]: What are you looking at?

Pembleton is playing cat’s cradle.

Bayliss: You have something that you wanna say to me?

Pembleton: Adena Watson. So many unanswered questions.

Bayliss: And you’re saying that I’m not asking them.

Pembleton: I’m saying that you’re not answering them. [He peers at Tim through the cat's cradle.]

Bayliss [sighs]: What questions aren’t I answering, huh?

Pembleton [gets up and flips through a notebook, draws a picture]: Okay, these sixteen row houses on the north side of Kirk Avenue. Adena’s body was found outside the kitchen door in the red yard at 718 Kirk. Now the killer could have dropped her anywhere. Why not the common alley? Why not the yards at either end of the block? These three row houses are empty. One, two, three. The killer would have stood much less of a chance of being seen if he’d dumped her body in any one of these yards. Why would he risk bringing a little girl’s body inside a closed fence of an occupied house?
Maybe he wanted her body to be found immediately. Maybe he wanted to cast suspicion on the people in 718. Maybe he had some … perverse sense of remorse, some impulse to leave her body inside an enclosed yard to protect her from stray dogs.

Bayliss: These are *exactly* [taps the notebook] the questions that I have been trying to answer.

Pembleton: Well, you can try, but you never will.

Bayliss: Why?

Pembleton: You don’t think like a criminal. You don’t have a criminal’s mind.

Frank walks away. Tim grimaces in disbelief.

Bang! Suddenly it struck me. “You don’t think like a criminal. You don’t have a criminal’s mind”.
In order for a police to think of possible outcomes of a crime they have to be able to think as criminals, and put themselves in a criminal’s way of thinking.
If you don’t do this, you are trying to understand and explain the crime based on your logic.
Similarly, in order to “attract” bugs you have to wanna see problems; you have to identify problems that might bug several kinds of stakeholders; you have to put all your knowledge about the project in to consideration; and you have to be able to see the problems that matter. You are not trying to understand and explain the system by using your own logic, instead you are using several input sources to do this: logic, subjective thoughts, people’s skill levels, complexity of system, technology, etc.

So let me present my take on a definition of “Bug Magnet”:

Being a Bug Magnet = Able to foresee possible problems (or able to spot opportunities for things to go wrong), in context.

The most important thing here is the last two words “in context”. That is, even if you have all the bug taxonomies and oracles in the world to support you, you have to be able to understand what matters in this project.
Knowledge about “all common problems in .NET applications” can help you sometimes. Knowledge about “all common problems in .NET applications that developer X often produce”, is however much more useful.
Understanding what bugs our users is more helpful than knowing about what bugs users in general.
Knowing about problems with focus in Windows applications is one thing, finding these problems during testing is to be able to spot opportunities when they are presented to you in your context (see http://thetesteye.com/blog/2010/11/windows-focus/ ).

Now, back to the questions in the beginning of the post.

• Is it luck?
  – No, even if luck sometimes help. For others it is more of welcoming serendipity.
• Is it faith?
  – I don’t believe in faith.
• Is it an ability that some are born with and some aren’t?
  – Maybe. Some people might have a more developed talent, but I think that most people have this talent. Some are born with variations of narcissistic personality disorders and might have difficulties with this.
• Can you learn this ability?
  – I believe so.
• Can you improve it?
  – Yes. However, you might need to consider one or several dimensions to improve: Empathy, reasoning, attention for detail and seeing the whole, recognizing patterns of your own and other peoples mistakes, subjectivity,  general systems theory, context-driven testing, and more.
  Notice that these dimensions are not technical but rather comes from social sciences.

This is a follow-up from Working with the testing debt – part 1 [1] and part 2 [2]. The reason for the clarification is that you so easily come up with a tip without a context or example.

Tip 3: Grow into a jelled team (read Peopleware [3] by Timothy Lister and Tom deMarco for inspiration). Peopleware identifies, among other things, things that you should NOT do in order for a group to grow into a jelled team. As a team it is so much easier to gain momentum in testing and especially so if you are jelled. Do you need to reconsider how you work as a team?

In one project I was test lead and tester. I had two fairly inexperienced testers assigned to me. I used a somewhat exploratory test approach to the planned tests, but they were far from organised and not well communicated. The tests consisted of a long list of one-liner test cases/test ideas. They were quite vague, thus enabling the tester to think for him-/herself. I had added some very specific test cases that were meaningless to run several times, but they were mixed in with the more vague ones. The testers thought it was meaningless to run the test cases several times, but I wanted them to do it anyway. Still, my intention was that they only use them as a guide. I did not communicate well and the testers did not feel like they were part of the test planning (which they weren’t) and thought I had set it up wrong. The mood in the team was foul. The testers had a hard time working with me for several years after that. Looking back I think they spent quite a lot of time that did not give any new information from what we had.

In another project I was assigned as tester together with my regular test group. We worked as consultants at the time. We did not know the project manager, test lead or any of the other testers that well. We were assigned test cases in different areas of the product suite. The test lead walked around and handed out test cases. There was no collaboration in the team. Sometimes when we talked over a coffee we noticed that we had done double work. Some tests were to be run in quite advanced configurations that took 90% of the time to setup. After a while we realised we spent 70-80% of the time setting up configurations, 10-15% of the time reporting bugs and the rest actually testing. By collaborating more we could have saved time to better switch tests around in the configuration that were needed between us. The team composition was 70% new consultants and 30% perm. employees.

And in yet another project the test team had worked together for quite some time. The whole team was involved in creating test missions/charters. Everyone tested and assisted in changing the test plan. When there was a major area that needs to gain some focused testing we assembled a group who generated test ideas, did some test design and did collaborative notetaking on the specific area. During the day the group debriefed several times and changed the test scope based on the feedback. Everyone was involved and felt great.

At one company we had assembled all testers into one big team (see [4] and [5] for setup) where we assisted all projects, the support organisation, the marketing organisation as well as taking on special missions from managers at different levels. When we started we had some unresolved conflicts, but as the team grew and as time went by the conflicts diminished. We worked hard together serving the organisation as best we could. As the team set forth we had the Black Team (inspired from Peopleware) in mind. We challenged the organisation to give us more to test, we could take on anything. We might have taken a bit too much pride in our work, but still I think that was needed after years of being looked upon as the nagging, complaining test team.

At the same company as above we sat in groups of four. When we got a new build we were armed to the teeth with test ideas, assigned areas for investigation and energy to shot down (not literarely) anything coming our way. During testing someone could shriek out in delight as a critical bug was found. The whole group would sometimes plung in and attack the same area from different angles to root out more issues, then continue on the track that they left earlier. We longed from the untouched (by testers mind) new features. We overwelmed the organisation with bugs, never having to consider that we might be the bottle neck at any time.

We all act differently depending on the context. With experience you hopefully learn from many of your mistakes. After you have been in a jelled team, you wish to be in the same situation again. You now know how good things could be and when you are not in that flow, you try to determine how it could be so. The book Peopleware is one of those things that might get you ideas on what you or people around you should stop doing  in order for your team to become jelled. Do not be afraid to tell management things that they that will stop you from growing. In some cases they might not know that they are doing it in a way that corrupts.

How does this affect the testing debt?

The benefits are obvious, a sentence to rise a cautioning finger about, but perhaps somewhat valid in this context. Part of the dilema is how you perceive the team that works well together. Do you see it is as a Clique (roughly described as an arrogant group) or as a Jelled Team? For my description I emphasises on a team that see itself as a jelled test team. When I say to increase the testing debt I mean that your backpack is getting heavier. Each time you start a new task, new project or new assignment you take the content of your backpack into consideration. Being a jelled team enables you to move more smoothly and become more flexible, thus able to become quicker and hopefully deliver better result.

We should also consider that there are differences between a team that is jelled consisting of various roles (such as a cross-functional team) and a team with only testers. Both constallations have different issues and both will have an ever growing testing debt. Some tips would apply to one constallation and not the other, naturally.

Kay Johansen and Anthony Perkins identified a few interesting ideas in their paper Establishing an Agile Testing Team: Our Four Favorite “Mistakes” [6] that I think is relevant to building a jelled testing team. We also have a paper by Elizabeth Hendriksson on Agile Testing [7] that identifies things you should not do as a tester if you intend to be agile. They are most relevant when considering what role testing should have in a cross-functional team or what your attitude is as a test team. Brian Marrick has made a compilation of articles [8] on Agile Testing, somewhat old ones but still valid as a way to consider when trying to become more effective and build a jelled test team. Marrick has also created the article Classic Testing Mistakes [9] that is valid still and should be considered when trying to identify why your team is not jelled yet. I’ve also written two blog posts about growing test teams: Uncertain team composition [10] and Progress [11]. Both tries to focus on things you should not do in order to get a jelled test team.

Some specific things that I see a jelled test team do, that decrease the testing debt, is:

• Build on each others test ideas. With this I mean that we are triggered on each others test ideas to create new ones until we have no further ideas.
• Easy of changing direction. I often promote the concept that the test team or the team in general need to be flexible, thus the naming of The Flexible Test Team. If the team works well and trust each other it will be easier to accept change.
• Collaborative notetaking. In a team where you want to work together and where you enjoy cooperation it is the teams effort that counts, not the individual. This is also true for taking session notes. This is no best practice (as none exists), but a practise that is good in some situations. That the team have the ability to do this is a bonus.

With an unjelled team, a group of people that do not collaborate as well as they could, thus increasing the testing debt, I instead see:

• Stacking of test ideas. This means that each individual identify test ideas and they are summed up, with duplicate ideas removed. An unjelled team is more inclined to do this than a regular jelled team.
• Double work and duplicate testing. This might be the result of bad leadership and test leading, but based on my experience I think it is more common than in a jelled team.

What things do you consider for a jelled test team or a jelled team that has mixed roles but with testers in it? What have I missed as you see it?

References

[1] Working with the testing debt – part 1 - http://thetesteye.com/blog/2011/05/working-with-the-testing-debt-part-1/

[2] Working with the testing debt – part 2 - http://thetesteye.com/blog/2011/06/working-with-the-testing-debt-part-2/

[3] Peopleware - http://www.amazon.com/Peopleware-Productive-Projects-Teams-Second/dp/0932633439/ref=sr_1_1?ie=UTF8&qid=1310020112&sr=8-1

[4] Flexible testing team - http://thetesteye.com/blog/2008/05/the-flexible-testing-team/

[5] Resource planning in a flexible test team- http://thetesteye.com/blog/2008/05/resource-planning-in-a-flexible-test-team/

[6] Establishing an Agile Testing Team: Our Four Favorite “Mistakes” by Kay Johansen, Anthony Perkins - http://agile2004.agilealliance.org/participate/examples/EstablisingAnAgileTeam.pdf

[7] Agile Testing by Elizabeth Hendriksson - http://testobsessed.com/wp-content/uploads/2011/04/AgileTestingOverview.pdf

[8] Agile Testing by Brian Marrick –  http://www.exampler.com/testing-com/agile/

[9] Classic Testing Mistakes by Brian Marrick - http://www.exampler.com/testing-com/writings/classic/mistakes.htm

[10] Growing test teams: Uncertain team composition - http://thetesteye.com/blog/2010/04/growing-test-teams-uncertain-team-composition/

[11] Growing test teams: Progress - http://thetesteye.com/blog/2009/10/growing-test-teams-progress/

This is a follow-up from Working with the testing debt – part 1 [1]. The reason for the clarification is that you so easily come up with a tip without a context or example.

Tip 2: Focus on what adds value to developers, business analysers and other stakeholders. If you do not know what they find valuable, perhaps it is time that you found out! Read Jonathan Kohl’s articles [2] on what he thinks adds value for testing.

In one project I worked with a group of experienced developers. I had not worked with them before close hand, but they had received some of my bug reports and knew me by reputation (whatever that was) in the company. When I got their first delivery to me to test I started right away. Immediately I got the feedback that I was not testing the right stuff and there were a bit chilly in their demeanor towards me. I investigated a bit what had happened and found out that they were not really interested in the minor bugs at that moment and that I should focus on the major issues. I explained to them that I report everything I find, I was not expecting everything I found to be fixed though. What was fixed was up to those who prioritized the bugs. Before I started testing I asked them what they wanted me to focus on first. After that they were a lot happier, both knowing I worked on things valuable to them and that they understood that I reported everything that I found.

During another project we were two weeks from the release of the product. We were in the middle of a transition from traditional scripted testing to an hybrid with both scripted and exploratory testing. Rather, we had test scripts that we used as a guideline when we explored, but we reported Pass/Fail on those. At that time Project Management was strict in wanting number of test cases run as well as the Pass/Fail ratio. Earlier test leads had not communicated well why these figures held no value. When we had run all planned test cases project management communicated to their managers that we were done. But we were not, we continued with working on our planned charters and ran sessions. We interviewed the support organisation, business analysts, product management and experts in the test organisation. Eventually we got a long list of risks and areas that we should investigate. We also got a long list of rumours that we intended to confirm or kill. Basically, we were far from done and we still had time before the release as we saw it. We had also received areas that people in the organisation found valuable to get information about. Still, we failed because we had not communicated enough to project management what we were doing. We managed to go through most of the areas and identified lots of new issues as well as killing many old rumours. We failed to bring value to some, but not all.

How does this affect the testing debt?

If you continue to work on things that have no value to you or any of your stakeholders, you must take a stand and change things. Do not accept the situation as it is. If you and everyone around you think you and your test team are not doing anything of value, it will just add to your testing debt.

As I state above Jonathan Kohl gives a good set of questions [2] for you to ask yourself to get back on the path. Also consider what Cem Kaner writes about in Ongoing revolution if software testing [3], because it is still ongoing and it not over.

References

[1] Working with the testing debt – part 1 - http://thetesteye.com/blog/2011/05/working-with-the-testing-debt-part-1/

[2] How do I create value with my Testing? - http://www.kohl.ca/blog/archives/2010_08.html

[3] Ongoing Revolution of Software Testing - http://www.kaner.com/pdfs/TheOngoingRevolution.pdf