thoughts from the test eye

When teaching scripted testing (yes, I actually do this!) I found the Pass to Fail, Fail to Pass heuristic (used by many, but now with a catchy name.)

The essence is that when a not-overly-simple test has resulted in a Pass, think about it some more, and try to make it Fail instead.
When a not-overly-simple test Fails, think about it some more, and try to make it Pass.
This will stop you from jumping to conclusions; you will find out if the essence of the test case was Pass or Fail; and you might have isolated a bug or two in the process.

Example: Test case is about resizing 200 images at once. The essence of the test (many images) actually works, but some testers might report a Fail, besause it didn’t work (but the reason was that (at least) one of the images wasn’t handled properly.) When Pass is reported, you might have missed a chance to run a complex test that could find important problems.

This is a specific instance of the generic Do Variations Heuristic (Creep and Leap is another instance) As with all heuristics, it is to be used with judgment.

Very soon the 1000th comment will be published on thetesteye.com.
Comments are our main reason for writing blog posts, because they take our thinking further.
Our ideas are challenged, taken in other directions, opening new possibilities (and closing some…)
Thank you!!

To celebrate this, we will reward the author of the 1000th valid comment with a prize.
It is not a big prize, so please don’t spam with “great job”.
We prefer comments that add value to us, and the readers.

This quantitative milestone requires a qualitative statement:
Comments sharpen thoughts.

I often promote that testers should sit close to or with their cross functional teams. Still, I am very fond of working in an isolated testlab environment where it is possible to shout, scream, play music and play out dramas that would otherwise disturb the regular office tasks.

The office landscapes that are open seem to be quite common. For agile teams, the environment can be setup so that everyone in the whole project is working close to each other. This is excellent in many ways, but perhaps not all.

I have, at several occasions, worked in a testlab which was isolated from the rest of the project or line organisation. During these times we were working closely with the developers, business analysts and product owners, they were never far away. Each of us had a space of our own where we could focus on tasks that needed no interaction. The testlab was where we had joint efforts for collaboration on test activities or experimentation.

Sometimes we used different types of music to affect our mood and indirectly the way we were testing. When we played music such as Queens of the Stone Age and Ministry it affected our mindset to be a bit more speedy, aggressive and non-forgiving. When we played music such as Nick Drake or anything bossanova, we instead took another direction. Still, all in the team had their own preferences for music and were naturally affected if they did not like it.

The testlab is a place where there is action, noise, interaction and collaboration. If you need to work on your own, that is probably not the place to be in at that moment. I believe that many projects need this kind of an area where it is ok to scream out in delight when you find a valuable bug or where the bass from the loud speakers beat with a steady rhythm while testers in their best way slam the system.

I confess, I am delightful when I find really vicious bugs. Sometimes I scream in dispair when the build is broken for the 50th time in a row. I like to express my feelings in the testlab, an area of free will and emotion. After this emotional disruption I can calm down and gather up the proper evidence so that I can present it to the stakeholders in a professional manner.

For those of you who have missed out on this, I urge you to create this area for creative test recreation.

Long live the isolated testlab!

Setup

Bart Knaack had done a wonderful job in setting up and organizing the testlab. He had done this by himself. So most credit to him for taking on most of the initial work and planning. Me, Ru Cindrea and Kristoffer Ankarberg focused on keeping the lab up and running as well as taking care of the participants. We wanted to be able to challenge experienced testers while at the same time help beginners get started.

In the lab the system to test were GeoGebra, osCommerce, OpenEMR, Freemind and Mindstorms LEGO with Mantis as a bug reporting tool. Mindstorms got the most attention, probably because it was different and so hands-on. The downside was that there were fewer bugs reported. GeoGebra was a new tool for the lab introduced by James Lyndsay. I believe those who tested it had a really good time, because I did. It was not as buggy as OpenEMR but it had equal amount of functionality with enabled lots of exploration.

Placement

As usual on the EuroSTAR conferences, the testlab is done through out the day where participants join in between breaks and at lunch. This setup makes it hard to facilitate and does not enable longer test sessions or longer ongoing events. Still, I believe it give the participants a constant flow of interesting events for EuroSTAR.

This time we were in the far corridor away from the sessions. Participants needed to go through the expo to get to us. Someone called it a gauntlet, for the testers to survive to get through to us. I think having the testlab on the far side made it better for the expo so that they got some exposure. Still, many of the sponsors showed tools that I do not believe in, but that is another story. The testlab was in an open area with pillars marking where its boundaries were. We had managed to get a printer by the EuroSTAR team, which enabled us to do so much more things for the participants!

Next to the testlab was coffee and lunch bars, this enabled many of the participants to grab some snacks and enter the lab or just watch while eating. I think this was a good move since it gave the participants a non-stop action. The downside was that the tempo was high and there was no rest for anyone, which might have caused a stressful mood to some.

Participation

Some of the more experienced testlab participants came to us in breaks and dug in to try our setup. The more inexperienced ones were a bit more shy, they needed a bit more guidance to enter. Once inside and sitting down, everyone showed great engagement. You could see a sparkle in their eyes as they found bugs or found something puzzling. This year it seemed like there was a greater mix of people from different approaches to testing. Previous years it seemed like there were more from the context-driven or agile approach to testing that visited the testlab.

The EuroSTAR star test team, that won the competition to be part of EuroSTAR, were in the testlab most of all. They were extremely focused in their testing and found some really interesting bugs in the systems. The EuroSTAR team wanted the star team to journal their stay at the conference using video. They interviewed many of the speakers and seem to have worked hard on their assignment. It was great having them in the testlab helping us but also by creating some activity just by being there.

Many people did an excellent job by bringing a college to the testlab and did some collaborative testing, sharing techniques and ideas on how to test. I’ve talked about this before and cannot say it enough times, this is a great way of testing and a great way of getting to know a tester you have not worked with before.

Sponsors

During sessions, when participants were most often listening to speakers, the sponsors demoed their products or services. There were hardly noone there and it did not work the way they probably had wanted it to be. Telerik used OpenEMR to show their product which was great, even if I was the only audience. I would recommend sponsors to engage with the participants in a different way. The current way of demoing is not paying off.

In EuroSTAR Testlab 2010 some of the sponsors were helping us taking care of the participants and if one of the participants wondered about a specific sponsor tool they were available to assist. The sponsors had installed their software on each client machine and had been working with our systems to show how their tool interacted and tested. I think that was a great way of handling sponsors.

Events

Simon Stewart held the keynote about Selenium. Prior to that he had prepared a few selenium scripts that he intended to talk about in the testlab directly after his talk. In the testlab, he had roughly 30 people around him that was engaged for more than an hour. I think this was a great success. Many who were at the conference were interested in automation and this was a key moment for them, I am sure.

One of the days we had focus on performance testing and one of the speakers held session in the lab for several hours. He had a small group around him that were engaged in talking and sharing experiences.

Markus Gärtner held a session on Testing Dojos where he moderated a group of testers to do test planning, testing and a debrief with reporting. I call this collabative testing, test planning and reporting. I use this setup in my every day testing as well as on previous test labs. I think it is a great way of working.

Bart and later on Michael Bolton showed a coin trick that is quite similar to the dice trick that James Bach and others use in their training. It is a way of training exploratory test design, reporting and testing. I was not able to participate myself, but it seemed like the participants had a great time and by looking at their faces learned a thing or two.

As the very last event in the testlab we had a test competition that Ru and Kristoffer organised on their own, to most extent. There were 8 teams participating who did a good job under the circumstances. It is a tough job to do something good when under time preassure, with limited space and in some cases having to communicate in a second language.
There were many more events in the testlab, but I was not able to be part of them all.

Reflections

I’ve already had a few reflections above about the testlab. Some that I would like to highlight more are that I would like to see more of Open Source Testing, something that Julian Harty talked about at Let’s Test conference 2012. It means that we make the artifacts from testing public, open for scrutiny and research. We collect statistics and data to be analyzed. We gather as much information and material as we can so that researchers and teachers can use it in their daily work to improve the testing community.

I would like to see tutorials and workshops have their base in the testlab instead of somewhere else. I believe the participants will have a great learning experience if we incorporated the theory from the speaker with the practice by the participants in the testlab. This would mean that we would need a bigger testlab or possibly a different setup of it. So a merge of one of the tracks that is more hands-on with the testlab. That is, if the testlab is done during the day. For Let’s Test we had the testlab as an evening activity, then the setup is different.

Having a printer in the testlab is a must from now on, I won’t live without it. We printed so much material and testdata that we would just cripple the testlab without it. I think a headset with mic would be nice to enable speakers guiding things in the testlab easier. We also need more whiteboards, flipcharts, pens of different colour, scissors, lots of tape and different papers of different materials, shapes and sizes. This is a workshop after all. By stocking the testlab with these items, we would enable more creativity and more fun for everyone involved.

If the testlab have sponsors, they need to be engaged in the testlab and with the participants. If the current setup continues they will just loose credibility as a company. At Let’s Test I had Compare Testlab as a sponsor, they helped me with keeping the server and wireless up to date and working. I could then focus on events in the testlab. Working together like that is a great way of making the testlab even greater.

The testlab is very much alike our every day life as testers. We prepare and plan for many things, but when reality hits us our preparations might be in vain. Therefore it is important in being prepared for the unknown and the unexpected. Working with the testlab is a great way of practising that.

Many thanks to the EuroSTAR team for helping us and serving our needs. Thanks to the Program Commity for being involved and participating in the testlab. Thanks to my colleges Bart, Ru and Kristoffer for being pragmatic and doing everything in our power to make this a great event for the participants. Finally, thanks to all participants who tested and shared their experiences with us.
Ru and Kristoffer will hold the testlab for EuroSTAR 2013 in Gothenburg. I am sure they will give you all a great time!

In a recent blog post [1] on Informator-blog, Magnus C Ohlson articulates the idea of pilots having the flight hours but not the actual flight certificate. He insinuates that the artifacts from requirements and testing would be better if people were certified, if I understand him correctly. Furthermore he explains that testers need education, knowledge and experience which I agree with fully. But he promotes that to ensure that someone has the right competence is by having a certificate such as ISTQB or REQB. According to him, you would then know if he or she is a skilled tester.

Here are a few personal experiences that relate to certification, as I see it:
When I did the military I got the opportunity to train for driving a trailer. We did a two-day theoretical exam, and then they let us out on the roads. After studying text books succeeding with a little exam they determined we were ready to start driving. I had no problem passing the exam, but I was a terrible driver. I had only a few months prior to this been able to take the driving license for driving a car. They let out a large amount of trucks, trailers and other vehicles into the small town of Boden. I managed to get around town without getting killed, but all in all the lot of us managed to demolish several traffic lights, some road signs, some trucks and a few light poles. The cost of this adventure was huge. I guess we learned a lot, but the idea that we were ready based on reading a textbook was a bit strange. Months later, after a few hundred miles of driving we started to get good at it, but the 2-3 day theory in the beginning was almost meaningless then. I see the test and requirement certification as having the same symptom, we do not get much value of something so intensive and theory focused.

In the early 2000 a doctor told me I had diaphragmatic hernia. At the same place, a surgeon went through with me the procedure how to get this fixed. She told me that I need to change the way I lived my life, but that it was almost pointless to try according to her. If I were to go through with a surgery, they would have needed to open me up, lift my chest and move around several organs. The whole procedure was very dangerous and the risk for death was high. As part of the surgeon’s analysis, she performed a gastroscopy. This procedure took between 10 and 20 minutes. It was hard for me to determine, after a while I understood what torture was all about. Some years later I met a doctor who hinted that there were other experts in the field that could help me out. I ended up with probably one of the best doctors in the field, who told me a whole different story. He recognized the name of the original surgeon who had given me the first statement. According to him, the original surgeon was not able to perform a gastroscopy in school as well. This new doctor performed a gastroscopy as well, which took 1½ to 2 minutes which is the time it should take, according to him. He then said that an operation was possible with minimum risk to my life. The last 10-20 years they had been using keyhole surgery, the methods the previous surgeon talked about are not used any more. Even if the first surgeon got her license, she was apparently not interested in keeping herself up-to-date with the changes in her craft. Comparing this with testing and requirement certificates, even if you are certified does not mean that you keep up with changes to the craft and that it is applicable.

A friend of mine was troubled about being forced to become certified. He was basically forced to certify himself as a project manager. The curriculum of the certification was solely based on waterfall methods. My friend has been working with agile projects for a while and has left the world of waterfall behind him. He does not see the point in keeping old facts up to date but instead try to learn new things. The person who wrote the curriculum for the certification did not know anything about agile. Comparing this with testing and requirement certification, we will have single point or points of failure in keeping up to speed with changes and improvements in the craft. If the creators of a syllabus are not top notch in the craft, then everyone who needs to certify themselves must lower themselves to the creators level. Even if they are top notch, they cannot compete with the wisdom of crowds, where the crowd is the joint knowledge of the test community.

Another friend of mine took an intensive course to drive a car. She travelled to a small town in northern Sweden to take a two-week course. She took it easily and got back home to Gothenburg. But the driving in the small town was a bit too easy with no highways, very few cars and an environment that did not match what was in her home town. She got the license and got home. When entering the traffic in Gothenburg she became too scared and dared not drive anymore. If we compare this with testing and requirement certification, an intensive course that results in a certificate might only mean that you have paid money to get something that is not valid in your actual context, in your project or hometown.

In a previous article called Testers Greatest Nemesis [2], I wrote about the intent that Dorothy Graham had when they initiated ISTQB. According to her blog posts the original intent was lost over the years. In Sweden there is a movement that has started certification of requirement experts called REQB. I see consultancies are offering the training and certification of it. I do hope this is just not a scam to make money, where recruiters will start filtering out those with 10+ years of experience of requirement handling to those with the name REQB in their CV.

Conclusion

I do not believe the complexity of organizations, projects and business is displayed in a generic multiple choice questionnaire that someone with no knowledge could accidently pass. I hope the movement in REQB learns from previous mistakes that have been seen by introduction of ISTQB. Dorothy and several who comments on her blog has identified a few things to consider.

A certificate tells one story about a person, but it is very fragile. I prefer talking to references, looking at their renown, public appearance, blogs, articles and papers would perhaps give a more vivid story.

References

[1] Varför certifiera sig inom krav och test – http://informatorutbildning.blogspot.se/2012/10/varfor-certifiera-sig-inom-krav-och-test_10.html
[2] Testers Greatest Nemesis – http://thetesteye.com/blog/2011/05/testers-greatest-nemesis/

We have now finalized the Swedish translation of 37 Sources for Test Ideas.

Neither the translation nor the original is perfect, but we think they can be useful to many.

If you are interested in translating 37 Sources, or Software Quality Characteristics (Swedish version), let us know.

Cheers,
Henrik, Martin & Rikard

Due to all spam comments not being stopped by our automatic checkers, we are unfortunately forced to moderate all comments.

We will do this as quick as possible, but there might be some delay before your comment appears.

Hope that you understand this, from our point of view, drastic decision.

Cheers,
Henrik, Martin & Rikard

I am curious about how the system works

I look at details and the whole, and at many places

I use many sources to learn what is important

I am creative and see many testing possibilities

I test in many different ways, and adjust to the situation

I am good at finding important problems fast

I am aware of models enabling and limiting my testing

I can evaluate a system from many dimensions

I can communicate my thinking and findings

I am an exploratory tester

Suppose you have a risk assessment fully agreed by stakeholders, and their relatives.
Create a stable sampling across all adjacent perspectives, and let the corresponding tests be executed on best representatives platforms, theoretically covering 99.5% of potential future usage.
Now, let any opposites conglomerate, and hold their best fit tempo, until the sounds dissolve into the night.
The information is tunneled into various status report templates according to confidently standardized information objectives.

No need for the Black Viper test technique anymore!

I hope many of you are using SFDPOT (James Bach) and CRUCSPIC STMP (thetesteye.com) in order to investigate what to test. SFDPOT describes elements of the product, and CRUCSPIC STMP describes sought attributes of the system. They are very powerful ways to identify things to test, plus to be able to communicate it effectively.

Both are very rich, and could incorporate each other, e.g. SFDPOT could be a part of Capabilities, and CRUCSPIC STMP could be a part of Operations (often inverted as a risk) There are more connections, as seen here:

I still recommend using both as separate activities, they stimulate thinking in different dimensions (and that’s why testers are needed, right?) That they blend is not a problem, it is the thinking that matters, so just put the stuff where it makes most sense.

If this makes almost no sense to you, interpret as a hint to try both!