thoughts from the test eye

Setup

Bart Knaack had done a wonderful job in setting up and organizing the testlab. He had done this by himself. So most credit to him for taking on most of the initial work and planning. Me, Ru Cindrea and Kristoffer Ankarberg focused on keeping the lab up and running as well as taking care of the participants. We wanted to be able to challenge experienced testers while at the same time help beginners get started.

In the lab the system to test were GeoGebra, osCommerce, OpenEMR, Freemind and Mindstorms LEGO with Mantis as a bug reporting tool. Mindstorms got the most attention, probably because it was different and so hands-on. The downside was that there were fewer bugs reported. GeoGebra was a new tool for the lab introduced by James Lyndsay. I believe those who tested it had a really good time, because I did. It was not as buggy as OpenEMR but it had equal amount of functionality with enabled lots of exploration.

Placement

As usual on the EuroSTAR conferences, the testlab is done through out the day where participants join in between breaks and at lunch. This setup makes it hard to facilitate and does not enable longer test sessions or longer ongoing events. Still, I believe it give the participants a constant flow of interesting events for EuroSTAR.

This time we were in the far corridor away from the sessions. Participants needed to go through the expo to get to us. Someone called it a gauntlet, for the testers to survive to get through to us. I think having the testlab on the far side made it better for the expo so that they got some exposure. Still, many of the sponsors showed tools that I do not believe in, but that is another story. The testlab was in an open area with pillars marking where its boundaries were. We had managed to get a printer by the EuroSTAR team, which enabled us to do so much more things for the participants!

Next to the testlab was coffee and lunch bars, this enabled many of the participants to grab some snacks and enter the lab or just watch while eating. I think this was a good move since it gave the participants a non-stop action. The downside was that the tempo was high and there was no rest for anyone, which might have caused a stressful mood to some.

Participation

Some of the more experienced testlab participants came to us in breaks and dug in to try our setup. The more inexperienced ones were a bit more shy, they needed a bit more guidance to enter. Once inside and sitting down, everyone showed great engagement. You could see a sparkle in their eyes as they found bugs or found something puzzling. This year it seemed like there was a greater mix of people from different approaches to testing. Previous years it seemed like there were more from the context-driven or agile approach to testing that visited the testlab.

The EuroSTAR star test team, that won the competition to be part of EuroSTAR, were in the testlab most of all. They were extremely focused in their testing and found some really interesting bugs in the systems. The EuroSTAR team wanted the star team to journal their stay at the conference using video. They interviewed many of the speakers and seem to have worked hard on their assignment. It was great having them in the testlab helping us but also by creating some activity just by being there.

Many people did an excellent job by bringing a college to the testlab and did some collaborative testing, sharing techniques and ideas on how to test. I’ve talked about this before and cannot say it enough times, this is a great way of testing and a great way of getting to know a tester you have not worked with before.

Sponsors

During sessions, when participants were most often listening to speakers, the sponsors demoed their products or services. There were hardly noone there and it did not work the way they probably had wanted it to be. Telerik used OpenEMR to show their product which was great, even if I was the only audience. I would recommend sponsors to engage with the participants in a different way. The current way of demoing is not paying off.

In EuroSTAR Testlab 2010 some of the sponsors were helping us taking care of the participants and if one of the participants wondered about a specific sponsor tool they were available to assist. The sponsors had installed their software on each client machine and had been working with our systems to show how their tool interacted and tested. I think that was a great way of handling sponsors.

Events

Simon Stewart held the keynote about Selenium. Prior to that he had prepared a few selenium scripts that he intended to talk about in the testlab directly after his talk. In the testlab, he had roughly 30 people around him that was engaged for more than an hour. I think this was a great success. Many who were at the conference were interested in automation and this was a key moment for them, I am sure.

One of the days we had focus on performance testing and one of the speakers held session in the lab for several hours. He had a small group around him that were engaged in talking and sharing experiences.

Markus Gärtner held a session on Testing Dojos where he moderated a group of testers to do test planning, testing and a debrief with reporting. I call this collabative testing, test planning and reporting. I use this setup in my every day testing as well as on previous test labs. I think it is a great way of working.

Bart and later on Michael Bolton showed a coin trick that is quite similar to the dice trick that James Bach and others use in their training. It is a way of training exploratory test design, reporting and testing. I was not able to participate myself, but it seemed like the participants had a great time and by looking at their faces learned a thing or two.

As the very last event in the testlab we had a test competition that Ru and Kristoffer organised on their own, to most extent. There were 8 teams participating who did a good job under the circumstances. It is a tough job to do something good when under time preassure, with limited space and in some cases having to communicate in a second language.
There were many more events in the testlab, but I was not able to be part of them all.

Reflections

I’ve already had a few reflections above about the testlab. Some that I would like to highlight more are that I would like to see more of Open Source Testing, something that Julian Harty talked about at Let’s Test conference 2012. It means that we make the artifacts from testing public, open for scrutiny and research. We collect statistics and data to be analyzed. We gather as much information and material as we can so that researchers and teachers can use it in their daily work to improve the testing community.

I would like to see tutorials and workshops have their base in the testlab instead of somewhere else. I believe the participants will have a great learning experience if we incorporated the theory from the speaker with the practice by the participants in the testlab. This would mean that we would need a bigger testlab or possibly a different setup of it. So a merge of one of the tracks that is more hands-on with the testlab. That is, if the testlab is done during the day. For Let’s Test we had the testlab as an evening activity, then the setup is different.

Having a printer in the testlab is a must from now on, I won’t live without it. We printed so much material and testdata that we would just cripple the testlab without it. I think a headset with mic would be nice to enable speakers guiding things in the testlab easier. We also need more whiteboards, flipcharts, pens of different colour, scissors, lots of tape and different papers of different materials, shapes and sizes. This is a workshop after all. By stocking the testlab with these items, we would enable more creativity and more fun for everyone involved.

If the testlab have sponsors, they need to be engaged in the testlab and with the participants. If the current setup continues they will just loose credibility as a company. At Let’s Test I had Compare Testlab as a sponsor, they helped me with keeping the server and wireless up to date and working. I could then focus on events in the testlab. Working together like that is a great way of making the testlab even greater.

The testlab is very much alike our every day life as testers. We prepare and plan for many things, but when reality hits us our preparations might be in vain. Therefore it is important in being prepared for the unknown and the unexpected. Working with the testlab is a great way of practising that.

Many thanks to the EuroSTAR team for helping us and serving our needs. Thanks to the Program Commity for being involved and participating in the testlab. Thanks to my colleges Bart, Ru and Kristoffer for being pragmatic and doing everything in our power to make this a great event for the participants. Finally, thanks to all participants who tested and shared their experiences with us.
Ru and Kristoffer will hold the testlab for EuroSTAR 2013 in Gothenburg. I am sure they will give you all a great time!