thoughts from the test eye

Preparation

The hardware setup of the testlab was 1 server and 4 laptops brought by Compare TestLab, 5 laptops by Adecco IT Konsult and finally 2 laptops brought by James Lyndsay. Many participants brought their own laptops. Before the conference I worked some on setting up a wiki, bug system etc. I tried to find a balance in not over-planning, but noticed later on that I had written down too little information making it hard for the participants in getting started and in knowing the overall missions.

By having a base of 11 laptops we were able to bring at least a few testers into the lab. Many thanks to Compare TestLab for sponsoring with both hardware and someone assisting us with the testlab environment, namely Torbjörn Wiger. Big thanks to Adecco IT Konsult for letting me utilize their laptops in the final minutes of preparation. Big thanks to James Lyndsay for being part of this confused setup and being patient with acting in the unknown and unplanned.

We had selected three applications as part of this setup, namely Sikuli, CiviCRM and Red Notebook. They were chosen because they were fairly buggy, open source and would make a good subject for interesting testing.

The planned agenda for the testlab had been published here with a bit more detail on what I had in mind here. During the conference the agenda changed and its content became something different, still within the planned concepts.

When we arrived to Let’s Test we started to setup things up almost immediately. I got help from Steve Öberg with some installation and by Torbjörn Wiger to set the server up. Doing everything on your own is hard so the help was greatly appreciated. I did not feel that we were fully ready when we stopped on Sunday night, but are you ever fully ready?

Day 1

During the day I talked to more speakers and participants to be part of the events in the test lab. I was very dodgy in details, since I only had rough ideas on what would happen. At 20:00 the door open to the test lab. Quickly afterwards Rikard Edgren, Simon Morley and Christin Wiedemann arrived and agreed to take a team and an application each as their focus area. The first event was collaboration test planning using Corkboard.me to plan in.

Participants started to arrive and we assigned them to the teams. Each of the team leaders did things differently based on what the team members needed. I noticed that some created missions/charters to be tested later while some wrote down their sessions directly. The content of each corkboard.me was different for each team depending on application/system and the team members. At the end of the collaboration session we held a debriefing where each team talked about what they had learned and experienced. Rikard, Simon and Christin held the groups together in an excellent way.

Start of the next session “Group Test Experiment”.

I held a short presentation about the various dimensions of group testing. I selected not to direct them in this, but left them to choose themselves what they wanted to try out. After a while I interrupted the teams and unfolded a scene/scenario that was about to happen. It went something like this “I, the test lead, am going to the release meeting for our products in 15 minutes. I want you all to give me the three most important bugs, where each team focuses on their application”. After roughly 15 minutes, I asked each of the teams to give me a 30 second report. Being able to give the essence of what is important is good to practice and often. Each team delivered some important facts all with a different pitch and undertone. The time was roughly 23:00 and it was time for a last beer before bedtime, at least for me.

Day 2

Discussion about having coaching session with Ilari Henrik Aegerter and Anne-Marie Charrett. Talked to more participants to get them into the lab. I let Rikard test some of the concepts for the evening event in the lab, and got some good feedback which made me change the agenda somewhat. Then I started setting up the lab so that it was possible for more teams instead of the initial idea of just two big teams, which must have been a good decision consider we were at the most close to 60 people if I read the tweets right. I selected Sikuli as the application/system for the evening event as the only system under test. A few people started to arrive at 20:00, but it was not that many to start with… so I considered if I had made the right move. All of a sudden the tables started to fill and the test lab was almost full.

I did a short presentation about the focus for the evening event, at least what I thought at that time. The idea was to have coaching of the teams in the first hour and later on to try out some group testing experiments. But I had to change that since the expectations of the participants were so diverse. Instead I said we would do a little bit of both during the whole evening, but with the major mission of creating a compelling status report.

Even more people arrived, who we filled up in the existing teams and by now we had 8 teams of 4-7 members (if I remember correctly). The context of the test lab had changed a bit from what my initial thoughts, I knew that the plan never survives when reality hits it. We had not put that much focus in describing in detail the full content of the test lab and how to get started in a good way. We thought that it would be easy to fill in that information as we went along, but that never became true.

Ilari and Anne-Marie came up with an idea for how they could coach the teams, which we tried almost directly. I told the teams that “HR (Human Resources) had decided that each team need to appoint someone to be tested.”. This act caused some confusion. Just prior to that I had told the teams that the focus was on creating a status report. So we had conflicting missions. Ilari and Anne-Marie did an excellent job even though it was a tough situation. Some teams were very focused on the testing mission and did not feel they had time for coaching, while others took it in and used the coaching to affect how the testing was done in the team.

Ben Kelly had joined in the test lab and I explained the current context and I explained roughly what I wanted him to do. I general I gave very thin outliers in how they could act in the test lab scene. Instead letting each player expand what was possible. Ben talked to the teams and coached them about what our mission was and about the creating of the status report.

During all this me and James were interacting with the teams. James gave some teams missions while I sometimes instead tried to confuse them. All-in-all we tried to simulate a situation where confusion and the unknown is our arena, but where me and James were roughly sure of what the other was doing in order to act on it.

After roughly 30 minutes of coaching by Ilari and Anne-Marie, I addressed the teams in the role of a project manager stating that “We don’t have time for coaching! I want those status reports!” The confusion was somewhat diminished and a renewed focus was visible. We decided that we would round up and hold debrief in about 40 minutes time, which was at 22:10. Some teams worked in X-Mind, another in the Corkboard.me UI, another in Post-It notes and yet another in Notepad. Each team focused on different areas, but some had found the same obvious issues. Some had found many issues and some found less.

It was my impression that Sikuli was quite buggy and that it really was not able to solve the automation in a good way without many workarounds. Many in the teams did not trust the application.

Even if we tried to keep the teams confused, that the team members where new to each other and new to the applications as well as to the test lab environment, they managed to produce quite good results. The traditional role of a test lead/manager could be questioned based on this or at least investigated further.

A few times when I confronted the teams and asked for specific results or information, where time was a factor, I noticed that some of the seemingly experienced context-driven testers returned questions, poking for more information. This is most often a good trait, but when time and timing is important you need to consider which questions you really want to ask that you believe bring value. Is this a common mistake that context-driven testers do? Still, a lot of the traps and confusion that was thrown on the teams was deflected in an excellent way.

Ben moderated the debriefing by asking for clarification then asked me in my role as a project manager if I was satisfied with the result of each time. It was a difficult question. From a test lab facilitator point of view, I wanted the teams to have a good time but also learn new things. I believe that objective was met. As a project manager I would probably had wanted to see a more consistent reporting style even though the information was on different areas. I was a bit split in what I actually thought here.

By setting up a scene it was easier to let the teams and their team members manoever in a changing context. It was very hard to balance between giving detailed instructions and giving a general outline in which they could act. Some teams embraced the scene quickly, while others found it harder. I believe it is important to be able to roleplay like this as a tester, to be able to act in a scene painted/described by someone else. It lets you explore so much more, with different mindsets.

During the Keynote of Julian Harty, he refered to “the testlab” but at this conference it was “a testlab” namely The Let’s TestLab, which was focused on learning, collaboration and experimentation. I am not sure we were fully doing open source testing nor any scientific method, is that still ok? I think we should embrace that many other testers run testlabs the way they want it to be. Otherwise we limit ourselves in what we can learn.

On previous test labs, I believe, the emphasis has partly been on the systems and applications. On this test lab it has been on the testers themselves and their collaboration. They have been able to explore some dimensions of group testing, note-taking techniques, collaborative test planning/preparation and finally seen different styles of status reporting. As a tester we know things could be different, but also that there is no one true way of performing a task. Hopefully this year Let’s TestLab showed that.

A big thanks to those who wanted to be part of the scene and acted in it. Another thanks to the speakers who helped facilitate with me and James. Finally, thanks everyone who participated that made it great!