Winning or losing in gamified test situations Martin Jansson No Comments
Games are not always about winning or losing. Each game can have different objectives. In my early youth I started with role playing games and later on story telling games. At the time, people who knew little about it sometimes asked us, “Who is winning?”. Being 7 years old, I didn’t really know what to say, in a good way, to explain to this grown-up that winning or losing is not always the objective. Instead, when we were role playing we worked as a group to go on with the story, to gain experience, to improve the character with skills and items and, probably above all, to have fun.
35 years later, I still hear the same question popping up in role playing situations, but also in similar situation such as when having to do with gamification. When we talk about gamification of testing, I feel it really demotivating if we were to compete against each other within the team or organization at a regular basis. I have already seen and experienced instances in my past when we had leader boards counting bugs in various fashion. We had really destructive discussions, as I see it, about if those who reported the least bugs really provided value to the group.
In a recent article  by Shrini Kulkarni approaches gamification of testing with the mindset of competition. I will look at a few of his arguments.
“This definition is provisional one – I might not be considering types of games that do not falling in this category. I wonder if there is any game where there is notion of victory or defeat.”
Yes, there are many different types of games. Believing that there is only games about winning and losing is too narrow. Some games are for introducing people to each other, others for passing time, others such as role playing and story telling games can be about solving puzzles/mysteries as a group, where you act as a different persona than your own you. The list is infinite.
“How about goals or objectives of a player or team playing games in the first place? Winning of course!”
Richard Bartle  investigated the objective behind different play styles in Massive Multiplayer Online Games (MMO) and Multi-User Dungeons (MUD) and discovered that only a fraction of the players had the objective of winning. Instead, there were other aspects, such as socializing, that were more interesting and in focus.
“How many times you heard the statement “it is not important to win or lose, participation and competing with ones best ability is important”. So if you lose do not feel bad – there is always another chance.”
In a testing context, working against others by competition would, as I see it, harm the organisation and the teams. As I stated in my previous article , when considering gamification you need to consider the regular traps of testing in order to work on areas that provide value. But you also need to consider how to get a good working environment. A competetive environment might not be the best solution? I do not see it fruitful to compete on many of our test activities such as information gathering. How do you weigh one type of information over another? Would you in order to win choose to not share valuable information to others in the team?
“A good test strategy in testing is same as winning strategy in games. But then – what is the meaning of winning the game of testing? Against who?”
We really do not have an opponent in our test strategy, as I see it. Still we can use many aspects from strategies of games and war in our reasoning. I often look for inspiration in the writing of Sun Tzu, Carl von Clausewitz and other strategists. Still, when considering strategy for testing I see it rather as meeting different objectives or goals for retrieving information, instead of winning.
Jonathan Kohl has taken elements from MMO-type-of-gaming and considered quests and adventuring . I believe this is an excellent area to look at. In most situations in role playing, the group cooperates working towards a set of goals or objectives, just like you would as a tester in teams.
So, for those of you who start to dig into the world of gamification in testing… do look beyond the regular winning/losing concept. There are more aspects at play here. Instead, see gamification itself as a complex system, which you in turn apply to other systems in order to enhance cooperation, motivation, feedback and learning among many things.
ISO 29119 – a benevolent start Rikard Edgren 3 Comments
When I test software I try to start friendly, to see the good things about the system, and to not focus too much on problems, that might not be important for the whole. So I did this for the new ISO 29119 testing standard, where I have read the three parts that were published September 2013.
Part 1 – Concepts and Definitions
The terminology definitions are much better than the ISTQB glossary. Less words, and not as rigid.
Also good and humble to omit multi-faceted keywords like quality, stakeholder, usability.
Happy to see my favorite testing heuristic included: Diverse Half-Measures (Lessons Learned in Software Testing, #283) – use many testing strategies in order to get less holes in test coverage.
Quality Characteristics look pretty good, and even include some Charisma in the form of Satisfaction (part of Usability.)
Part 2 – Test processes
I love that ISO 29119 puts a lot more focus on test strategy. It comes before the test plan, which means that key strategic decisions will be communicated early. This not only makes sure focus is on important things, it also makes future reporting easier, since testing objectives are anchored.
I hope this puts even more energy to the current wave of test strategy awakening.
A key benefit of using the standard is that clients will know which documents to review, and where to look carefully. Especially useful since the adherance to the standard can be tailored, it supports both Full and Partial Compliance.
Nice to see that results from test execution isn’t totally binary: Pass, Fail, and unexpected.
Part 3 – Test documentation
The documentation standards also have examples from both traditional and agile projects (the standard duly notes that the example content shouldn’t be used as a template to follow.)
An example for dealing with agile projects: it is perfectly fine to not report status in written format.
I think the test plan will get most followers, and it is indeed a hotter version than IEEE 829:
How about ensuring that testers think about their stakeholders and how to communicate with them!
Also a plus for suggesting visual representation in the usually text-savvy test plans.
A much awaited separation of product and project risks, since we deal with them differently.
Some of these important things might be forgotten otherwise…
So is it useful?
Well, too early to tell, this was just the benevolent start…
Project managers have a great impact in our daily work and can often affect how we work, thus what is meaningful or not.
Several years ago I was involved in an organization which was split on several sites in different countries. Project members were not co-located, instead they were split up on different sites. Communication were often a bit strained and confusion because of cultural differences were not uncommon. All test teams reported to project management, who operated from one site and managed testers both at their own site and at other sites.
The project managers wanted to somehow know that testing had good progress. The traditional test process used test cases as a way to report progress. Traditionalists in testing had recommended that the project managers asked questions to the test teams on how many test cases had been executed each day and what the pass/fail ratio was. I believe this is a common, yet dangerous, recommendation.
One of these test teams did not executed as many test cases as the project management wanted. It seemed to them that the test team did not work as much as they should. So, the project management put some pressure on the test team to run more test cases.
The same test team had gotten a new manager of testing. At this time, the new manager contact me about being worried about how his team of testers were conducting their testing. He explained, in order to show progress to the project management, the test team rerun the same test cases several times on the same system with the same version of the system. When the test team reported to project management, they in turn were happy with the new, promising result of a high count of test cases and their pass/fail ratio. The project management was not aware of that the test team were rerunning the same test cases several times. They only looked at the presented metrics and ratio, no details behind it.
There were many problems with this situation. One of them was “you get what you ask for“. Asking for a high count of test cases and promoting it, will result in a high count of test cases by whatever means possible by test teams. Another problem in this case, is that when we talk about progress of testing they did so in terms of test cases executed. The daily activities in the test team does not consist of only running test cases, instead there are so much more that is being done. Yet another problem is the idea that rerunning the same test cases on the same version of a system would yield a different result.
How do you affect this situation to turn it around? The manager of testing who contacted me, tried to coach the testers in doing things differently by going beyond test cases and the execution of them. I discussed with project management about the current situation, shedding light on what result they got based on the questions they were asking.
A test team have a lot of different activities. Testing or executing test cases is a mere fraction of what we do as testers. Still, our main activity should be to test, in most contexts. To talk about progress and just listen to what testing we do, does not give a full picture. Instead ask progress on test activities and if anything is blocking us, or perhaps if we need help with anything.
Health of the system
A separate question should be about the system, sub-system or whatever that the test team is testing. What is the current health of the system? What issues or bugs do the test team see? What major risks that we did not know about before? What areas are now known, but that we still lack information about?
Information about the system is continuously changing. As new versions of the system are produced, earlier information degrade or might not be valid any more.
If progress is asked on test coverage, remember that coverage is linked to models of the system. As testers we employ many, many models and each have their own coverage. We can show progress in the sense that we can show what we know or think we know. We can also show areas that we want to know more about, that we currently know too little about to be able to show anything coverage related. We have questions to the system, the project or a situation that could still be unanswered, we have no idea what it leads to or what is hiding behind it. So talking about coverage in terms of percent becomes a bit absurd in that sense.
After having a long discussion with the project management, it was time to see what changes were to happen. The main project manager directly started to use a different language in his questioning instead focusing on progress, health of the system and coverage. As a bonus, he wanted the testers to explore the system beyond what we currently knew.
The result was astounding, seeing change in motivation, test result and amount of information produced from testing. As I see it, the new way of working followed methods that are proposed by many testers with knowledge in exploratory testing, thus mostly non-traditionalists.
Using gamification to explain and model testing Martin Jansson 1 Comment
In early 2013 I held a 7 week course on setting up a testing organisation that works well in an agile context. My intent was to explain my own approach and model of how testing is conducted, I wanted the students to see that they needed to create their own. For each part of the course there were exercises for me to evaluate the students knowledge and skill, they also needed to explain testing to me, with their own words and therefore using their own models.
I had one exercise that I wanted the students to do, but in the end I did not let them which I regret. I wanted them to gamify  testing. My idea was that by gamifying it they needed to explain all the intricate details of testing. They needed to identify activities that was considered valuable and motivating while at the same time identify activities that would be wasteful or meaningless. The students would need to model what testing is for themselves and explain that using gamification.
Jonathan Kohl has written many great articles on the gamification of testing  and has in a specific piece elaborated around the concept that Software Testing is a Game . He identifies several aspects that need to be considered when applying gamification to testing. Part of my reasoning is inspired by my brother Ola Janson, who has been in the gamification domain for a long time.
This is roughly how I would suggest to go forward with this as an exercise.
Initially, you would need to identify what tasks and activities that we do in the testing domain. This part is a great opportunity to visualize and clarify what you believe you do when testing, how its parts are connected and what you found valuable. You will or at least should identify things that you probably do not find meaningful to do. An important part is also to categorize and group the tasks and activities in order to enable interesting modelling. Examples of these models are the Heuristic Test Strategy model  and the one that Jonathan Kohl has in Demystifying Exploratory Testing .
Next step is to create a framework with basic rules, challenges and rewards that would lead to meaningful, motivating choices.
Finding challenges is easy, but finding resolutions for them is hard. It is difficult to look at the full scope of testing, instead it might be better to start with one part. In a recent meetup focused on gamification in testing, we discussed what part to select and agreed that regression testing could be a valid part to start at. Everyone had ideas on how to make it more motivating. In many cases the discussion was about that the participants did not want to do regression testing in a way that was meaningless. The group identified several things that could be done to avoid doing tasks in a meaningless way. We considered how to add gamification mechanics to make it more motivating. One reflection that we had was that a lot of what we thought we could gamify had to do with feedback to others or from others, information sharing and basically about improving communication. My own reflection from the meetup was that it was possible to use gamification to communicate value in testing with people that might have different ideas on what testing was.
When working on rewards they would need to consider dangers of metrics , automation snake oil , automation politics , biases and fallacies  among many things. By taking all these traps into consideration when applying gamification, I believe they would better explain their own model of testing. Things such as bug top lists (ladders), counting tests or something similar would for instance be a demotivator instead.
My thesis is therefore that by letting someone gamify testing, they need to have great knowledge and skill in order to create a plausible model of testing. I have my own approach and model of testing that I am now gamifying in order to hone and sharpen my ideas even further.
How would you gamify testing?
Are you able to visualize your own model of testing?
Kahneman and Test Strategy Bias Rikard Edgren 4 Comments
Our minds are fantastic, but can be biased and make dubious decisions. As I read Kahneman’s Thinking, Fast and Slow, I thought about test strategy and found some interesting stuff, based on mistakes I have seen, and done.
Answering An Easier Question
“How could we possibly test everything important?” is a very difficult question. It is natural, but not necessarily good, to exchange with a question that is easier to answer:
- How should we test the new functionality?
- How should we do the same testing as last time, but faster?
- What can we automate?
- How long does the test plan need to be?
When we answer the simpler question, we subconsciously think we have answered the difficult question…
This can also happen when we are asked to estimate how long something will take to test, before we have considered which strategies that are appropriate. Later on, we will probably choose the strategy that will give the result from the estimation. Another version is to split up the problem in smaller pieces, e.g. test phases, where responsibility is distributed, resulting in loss of the whole picture, and important areas uncovered.
Cure: Seek the difficult questions. For each part of your test strategy, consider if the answer (or the question!) is too narrow. Especially look for opportunities where a specific strategy can be used for more additional testing missions.
Example: When performing manual testing, change platforms and configurations as you go, so you won’t need a special compatibility phase. When appropriate, do spot tests for additional platforms.
What You See Is All There Is (WYSIATI)
It happens that the strategy only consider the testers’ (and developers’) testing, when other strategies might be more powerful. But also the opposite happens, that customers are doing acceptance testing doesn’t automatically mean that no one else should consider if the product is “good”. A common mistake is to only “see” the requirements document, and believe those are the only things that are important to test.
Cure: Discuss what else to test, not only what is inside the project plan.
If something is good or bad, we tend to transfer those attributes to other, unknown, areas. As an example, a long and handsome test plan can make us believe that the strategy is reasonable; and a sloppy bug report make us doubt the content. A negative effect happens when the behavior of the product on one area effects our expectations on other things. When something looks good, we might base our strategy on the belief that the rest also is good; if we see a crash at once, we might think that further testing is pointless.
Cure: Don’t build important decisions on single observations; look more, don’t have prejudices about good or bad product quality (let the observations judge.
Counter-example: Distribute your strategy in plain notepad format, so the content is in focus.
Illusion of validity
You run the same strategy as last time, because you found important bugs. That equally many were missed, and that it was very time consuming, is disregarded. One example can seem to make a strategy valid, 1 important issue after investigating 100 backward compatibility files make it worth the effort (which might be true, but there might be other ways that would be better.)
Cure: Even experts can’t predict the future, so make sure to have diversity in your test strategy.
Example: Since we have run free exploratory testing for our final regression tests for the last releases, we will this time do it function by function.
Our plans tend to be optimistic; we think all days will be good, downhill with the sun and wind supporting us. This also happens to our testing strategies, they aim higher than we can reach, especially since the unexpected always happens. That’s why it is important to communicate which parts of the strategy will receive most time, and which parts will be tested lightweight (it is seldom wise to skip relevant parts totally.) Your strategy should be realistic, and consider excessive optimism and unknown unknowns.
Cure: Make sure you communicate your priorities, so the less important parts can be skipped, or tested shallowly.
“No separate part of the testing strategy is as important as you imagine when you think about it.” I still get trapped by this one, especially when I try to convince someone about a certain way to test. The truth is that a lot of test coverage for the most important stuff is overlapping; a severe installation problem is captured by any test method, an alert manual tester can easily see usability, performance and security problems, and the specialists for each area can also look broadly.
Cure: De-focus. Look at the whole, discuss with people not obsessed with you current area.
Regret and responsibility
An important explanation for dubious test strategies could be fear of regret and blame. If you run the same strategy as last time it is less risk for complaints than if you actively changed the way you tested. If you can reference a best practice, people might think that reasonable choices were made. But as with other biases, one should rather choose what one thinks is best, than keeping your back free.
Cure: Diversity in the test strategy, and courage to change when you see the results.
Example: An automation strategy hasn’t produced much in one year. Add more resources or start all over?
Bias is a natural people-thing that often helps us. It can’t be avoided, but it can be managed (by being aware of it.)
In his book, Kahneman often mentions luck, which is important also for testing. But good testers are often lucky, and a good test strategy creates opportunities for good fortune. In a sampling business, serendipity is nothing to be ashamed of!
What Is a Good Test Strategy? Rikard Edgren 5 Comments
To me, a test strategy is about “what to test and how” and “ideas that guide testing towards the testing missions“.
It is an ongoing process, not necessarily a document, and important for the results of the test effort.
A good test strategy should be unique for the situation, but there are som general properties I try to reach:
- specific – details rather than fluff
- practical – possible to execute with “normal” turbulence
- justified – reaches the testing missions
- diverse – important systems needs to be tested in many different ways
- resource efficient – uses available resources without (too much) waste
- reviewable – possible to understand and review, so it focus on right things
- anchored – in management, in testers
- changeable – to be able to deal with the unevitable unknown
- erroneous – if it isn’t “incorrect”, it is too vague, or took too long time to write
In the real world, I believe the “reviewable” part is very rare, which also means it probably isn’t anchored.
Reference: James Bach, Test Strategy (includes the 3 first properties above, and a very good example)
10 Years of Lousy Test Strategies Rikard Edgren 7 Comments
For the first 10 years of my testing career I wrote lousy test strategies. I believe the actual test strategies, what we tested and how, were adequate, but the way it was communicated, as part of test plans, was not good. As many strategies, they more or less just stated different functionalities, and that they would be tested. They were too easy for stakeholders to skim and say “seems reasonable”, without critical thinking.
My turning point came when I read a strategy of Henrik Emilsson, where he explained investigations about what was important; where he in an easy-to-read manner captured how the important stuff were to be tested. It was written so that stakeholders could review the strategy, and make suggestions about how to improve it. Not only does this produce better strategies; it also anchors the strategy, and makes forthcoming reporting easy, and effective.
I have learned my lesson, and nowadays I dare to tell in which ways we intent to test, I mention names and tools, I expose my interpretation about all quality attributes I think matter. I know I might be wrong, but I want to be corrected as early as possible. I still don’t want to promise how long things will take, but I inform about what I think will take most time, and I don’t hide areas were I see the biggest challenges. I don’t write more text, I just focus on the important stuff; knowing that the conversations around the strategy might matter most.
Robin Hood Test Courses Martin Jansson No Comments
James Bach has written about Sweden and its testers like this:
“In my world, ‘Swedish tester’ is becoming a stock phrase, like ‘French chef’ or ‘Swiss banker’ or ‘Antarean starship captain’ (You want your hyperdrive fixed right? Go see an Antarean). I’m not entirely sure why this is, but part of it is their cosmopolitan, egalitarian culture. Skilled testers find more fertile ground, in Sweden. ” .
It is an inspiring badge we have received. I hope this is one of the reasons that EuroSTAR again takes its place in Sweden. How do we maintain this reputation? I think we have to accelerate our learning and look beyond corporate profit. The Queen in Alice in Wonderland puts it interestingly:
“Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that! ” .
There are a lot of things we can do to improve our skills and knowledge in testing. I myself learn a lot by blogging, writing articles, and by putting theory into practice on consultant assignments. When you write, you have to do research and build arguments for your claims—you have to immerse yourself in a topic. I also attend courses, workshops, and conferences as often as I can. Unfortunately, the cost of most conferences and courses are high. In Sweden, many test courses seem to cost between 5.000 to 10.000 SEK (Swedish krona) per day—I think this is too expensive.
Many businesses suffer from tight budgets that do not allow for their employees attending courses—especially not on several occasions. To summarize, the current situation is risk-heavy for course organizers and expensive for participants. I do not think it needs to be like this and I want to do it in a different way. Here are some things I want to try to improve in Sweden with courses:
- Make it easier and cheaper to attend a course.
- Offer a set of diverse courses in testing.
- Enable trainers to try new ideas in testing through courses.
- Make it easier for trainers to let new courses be available to the public.
- Enable training providers to try out unfinished or experimental courses.
- Better take advantage of digital social networks to work with ideas around the course and with other course participants.
It may sound a bit optimistic, but there are already tools for social networking to support, at least in part, my proposal for how to organize courses. What is the problem with the current situation?
Current situation with courses
Being the organizer of a course or conference involves investment in both time and money. Money that you eventually want returned. As an organizer, you are not sure if the interest is high enough—if enough people will sign up. Then you have fixed expenses such as the lodging, travel, food and other items. You need a classroom and food for the participants. In addition, you probably need to provide a flat fee for trainers. You may need to pay a deposit, and quite quickly you will notice that costs are climbing. There will of course be less risk and expense if the organizer has their own trainers or employees who hold classes, i.e. no external trainers. If there are few who participate, it will yield little result, but if there are many participants, there is a lot of money to be made.
Here is an example of the various costs associated with a course:
The table above consists of several columns; the first naming the area, the second to fifth comparing between different amount of participants. Cost of food varies dependent on amount of participants, while the rest are independent of participants. In this example, the income for the organizer is 24.000 SEK for each participant, which means that the more participants there are the better profit will be made. With a course this expensive, it will be enough with six people for it to break even. If you get a full house you could expect a profit of almost 500.000. In many cases you can expect more cost such as expenses for advertising and the like, which obviously lowers the profit.
How can this be setup differently?
The organizer will contact the trainer for a price. An estimated total cost including organizers fee plus any other cost for the course. Let us suppose that we make a similar calculation as above example with the same price for the course. Though this time we do a calculation in which participants share the cost.
The table above is setup similarly to the one prior to this with the exception that profit is zeroed and the introduction of something called shared cost instead of income. The organizer gets their payment for the expenses and hours without any profit. The price for the course drops if there are more participants. As you can see, it is enough for six participants to get a cheaper price than when you go for a more traditional course. The price, with shared cost, is really low if everyone can come. There is almost no point to arrange the course if you are below a certain number of participants. It is also possible to reduce costs even more by keeping the course in a free classroom, if food is not included, if you involve sponsors and so on.
The next step is to spread information about the course so that participants can start registering. This is to see if there is any interest, show the estimated cost and what the approximate price will be. For this you can use a tool such as meetup.com, for planning and as a main tool for networking. When you schedule a meetup, there are a lot of parameters that you can tweak. For example, you can specify how many people can come, if there is a waiting list, the price and how to pay. Beside this, you have a variety of other settings you can use to simplify your meetup, course or event. At this stage, you should have most of the information available and be ready to host the course.
The next step is to agree on the date. For this you can use a tool such as doodle.com. You list a number of dates which participants can choose from, then each participant clicks on the dates that they can participate. That way you’ll find out what date is best for everyone and can thus maximize the number of participants. Do not forget to possibly let the course be on a weekend, since it can be preferable for consultants in some cases. The organizer should by now have an idea about when the most participants would be able to attend.
The organizer should now be able to charge for the course with a shared cost model. Furthermore, you should also find a classroom and others such details, i.e. things that take time to organize. After that the course should be ready to be held.
For most courses, there is always something to prepare for the participants to increase the learning experience. There might be texts to read up on, software to become familiar with or possibly to install. It is also convenient to create email groups and somehow tie the participants together. This is to increase networking and learning, but also to enable for better group work. When the course starts participants should be prepared and energized! If you booked a hotel outside the city for the course, the networking and experience sharing will continue late into the night. All of these networking and preparation activities the organizer could help out with.
If a participant is unable to attend the course, they have always the option to send someone else or give the spot to someone on the waiting list. It will be difficult with this kind of approach to reimburse when someone cannot attend, but if there are creative suggestions it can be resolved.
In regular courses, it ends with the course being finished, but in our case we want participants to continue to network based on what they have learnt. They may wish to try out what has been learned in their own context then share with others how they plan to proceed, and what obstacles to look for when introducing changes. By networking both before and after, I think we can gain more from the course itself. This is something that the organizer can try to enable.
I have already scheduled a number of upcoming courses with this model of shared cost. The initial goal is to see if this is possible. Anyone, even people outside Gothenburg, who think this sounds like an interesting setup is welcome to attend or set it up in their neighborhood. Initially, there are mostly foreign trainers, but some local Swedish trainers are on the way. You can find more information on our local meetup Passion for Testing . Beside the current lineup, I want to take this even further.
Untested test courses
If you have a testing course that you want to try on a group of testers, participants of the shared cost model might be a good audience. Personally, I want to experiment with the theme agile testing organization and collaborative testing.
Participant powered test courses
There will be opportunities for participants to identify needs for courses, where we can look up and take in trainers. There are a number of experts in areas that are interesting to look into that have no official courses of their own. Examples of this may be an intensive course in a specific test tool.
As you probably realize that there will be a lot of work for the organizer while at the same time it becomes less profitable, but on the other hand less of a financial risk with the shared cost model. This setup provides an incentive for participants to get more to participate because their own expenses will be lowered. The amount of job for the organizer will be almost as much independently on how many will participate. In a dream scenario for a participant, one could attend a course that is a bit more expensive course and then a couple of less expensive courses, all in one year, while staying below the yearly budget for courses. It should also be possible to introduce longer-term investments to even better mitigate risks.
The next area to look at is how to get to this for testing conferences.
 Recommended People by James Bach – http://www.satisfice.com/recommends.shtml
 The Red Queen’s Race – http://en.wikipedia.org/wiki/Red_Queen ‘s_race
 Passion for Testing – Meetup – http://www.meetup.com/Passion-for-testing-Goteborg/
Translations of Quality Characteristics the test eye 5 Comments
To our delight,, Software Quality Characteristics have been translated to five languages.
Not only have these people created more words describing aspects of quality, they have also developed their own understanding.
Dutch – Software Kwaliteits Kenmerken (Huib Schoots, Ruud Cox, Ray Oei, Zeger van Hese, Jean-Paul Varwijk, Jeanne Hofmans)
Polish – Charakterystyki jakości oprogramowania (Radosław Smilgin)
Romanian – Caracteristici de Calitate a Produselor Software (Adina Moldovan, Ramona Tripa, Alexandra Casapu)
Norwegian - Kvalitetsegenskaper for programvare (May Britt Sandtorv, Geir Gulbrandsen)
Swedish – Kvalitetsegenskaper för programvara (Rikard Edgren, Henrik Emilsson, Martin Jansson)
Translations to other languages are encouraged, drop us a mail if you are interested.
Henrik, Martin, Rikard
Reflection from Let’s Testlab 2013 Martin Jansson 1 Comment
Planning a test lab takes a lot more time than you think. You prepare a lot of things and try to make the event great. Here are a few things that I considered …
What applications/systems to test?
- how many would be enough?
- what type of system?
- how big or complex?
- are they fun to test?
- do they have enough faults in them?
- are they open source?
- is the project/company interested in feedback?
- has the system been used elsewhere in test labs recently?
- can we gather enough test data or other material for it to be testable?
- would these trigger interesting discussions?
How do you share information about the test lab?
- wiki or web page or something else?
- does anyone read anyhow?
- how to report a bug is probably a good idea?
- does testers with different skill level require different information?
What is your schedule and how are events set up in the test lab?
- have you gotten hold of any of the speakers that want to be part of the test lab to extend their talk?
- will the speaker show up, you do get tired after a talk?
- depending on the conference, the participants will be receptive to different things
- depending on the conference, the main schedule will be different
- if the test lab is done during the evening, it should probably end before people fall dead
- what other sessions or events are performed during the test lab that you might want to sync with
What venue assistance do you get for the test lab?
- can you get print outs during the test lab in any format, size and color?
- do you gain access to white boards, flipcharts, pens, scissors, tape and papers?
- do you have power and cords everywhere?
- can you present in the test lab in two places?
- how many can be in a room without violating security or restrictions for safety?
How have you handled sponsors to help with the test lab?
- do you have a sponsor that handles all the client machines?
- do you have a sponsor that handles the servers and wifi?
- do you have sponsors that want to install tools?
- do the main conference sponsors have specific requirements for the test lab?
- have all sponsors installed their tools on the client machines?
- will the sponsors participate in the test lab and help out other testers during the events?
- are the sponsors aware of the expectations in the test lab?
- is it possible to have sponsors in the test lab?
- what shallow agreements do you have with the sponsors?
How do you intend to handle bug reports for the various systems in the test lab?
- will you set up categories for the bugs so that you tailor where they report bugs?
- will you leave the categories open so that the testers have more freedom?
- will you use a separate bug system such as bugzilla or mantis?
- will you use a solution such as Redmine or Trac to handle wiki and bug system in one?
- will you have a few bugs reported before hand to help guide participants?
- will you review bug reports and help participants when reporting?
- will you when finished report all bugs to the project owners of the systems?
How do you handle information from the projects and owners of systems under test?
- do you ask for what they would think is valuable?
- do you ask for a mission for the testers?
- do you ask for their fears, risks or rumors that they wish investigated?
- are you at all interested in what they think?
How do you handle builds and versions?
- do you set up oracles such as earlier versions?
- do you have nightly builds?
- do you have a recommended version on a USB?
How do you handle existing information about the testing of the systems under test?
- do you gather session notes, mind maps, test matrices or some other kind of artifact?
- do you gather models of the system and coverage models?
How do you handle test data for the systems under test?
- have you set up test data based on a domain analysis?
- have you structured the test data for ease of use?
- have you documented the test data so that testers of different expertise can understand and use it?
- have you test data to perform load tests or performance tests?
How do you organize testers in the test lab?
- do you let them just sit down and test randomly?
- do you try to gather them based on specific missions or skills?
- do you let teams form on the spot?
- do you have predefined teams that have booked spots?
- do some speakers have booked spots for others to join up around?
How do you handle pins, prizes, awards or rewards?
- do have your regular set of test lab pins to hand out?
- do you have specific prizes for specific parts of events?
- what do you promote in the test lab that you would give an award to?
- are you inspired by spirit of the game award from Ultimate Frisbee?
- do you have any fake certificates to hand-out to promote a special ability or skill?
- do you have T-shirts or any free give-aways that can be handed out?
How do you handle bells and whistles?
- do you promote participants making sounds when they find bugs?
- do you promote focus and silence instead of interrupting sounds?
How do you work with your partner[s] in the test lab?
- do you split things between you?
- do you cooperate on everything?
- do you have a day each?
- do you have a partner?
How do you handle artifacts generated from the test lab?
- do you follow Open Source Testing principals by storing the artifacts in a public repository?
- do you save them for the next conference?
OK, now you see a few of the things we consider before the conference. So, what happened at the conference?
James Bach talked, among many things, about something called “Shallow Agreement”. This is the first thing I experienced regarding lab setup. One of the sponsors had sent me 15 laptops that we could use in the test lab. Perfect! What was shallow with our agreement between us was what a tester do with a laptop. I will not, hopefully, do the same mistake again. What I should have clarified was that I expected the laptops to have full administrative access. Tester want to install, uninstall, reinstall, monitor applications, install our favorite tools and share them among fellow testers. We probably want to use our favorite editors and some probing tools. We probably want to be able to start and stop services, kill any process and change the behavior of the system. Basically, we want to be admins on the environment we are testing. Without that we are blocked. So, the next morning I sent all laptops back to the sponsor since I was not able to gain admin access because of sysadmin policies. It was a stupid mistake of mine to not make it clear to the sponsor what I expected and how we should operate the laptops. Since the sponsor was a test tool vendor I just presumed that they knew. But no matter what company you work with there will be shallow agreement that you need to identify and eventually avoid.
Now the first day of the conference started. Me and James planned out the details for the schedule of the first night and imagined that with lots of tweets people would bring their own laptops. We were just going to finish up the setup real quick in the test lab so that we could join the sessions and talks. 7 hours later we ate dinner. Then we were almost ready. Compare Testlab, the sponsor who setup the server and run the wifi, helped us in an excellent way. Torbjörn Wiger from Compare managed to keep his calm and helped us throughout the conference to the last day. We were so happy that we had that sponsor aboard. This way, we could focus on the event and activities in the test lab and instead try to minimize the activities around the equipment.
4-5 teams had registered for the test lab, we were ready for them! Clock struck 8 and the test lab was open. The test lab was empty, apparently many got stuck in the bar. So, our nemesis for this evening was the bar. As people started arrive they saw that there were no laptops, some arguments arose about the lack of laptops. Yes, I know… it would have been excellent with laptops, but only if they had admin access. Teams started to arrive and were trying to setup their laptops, install test data and understand what they were doing there. Apparently many had missed our tweets about bringing laptops. Oh well, so much for information overload.
I didn’t release how angry I was about the laptops until I started to introduce everyone to the test lab. It did not help that I argued about the laptops, or lack off them, with one of the testers. I started blabbing and blundering something, then gave the word to James Lyndsay who took over without a sweat. James facilitated the test lab the first evening, while I went around trying to help the best I could. Note to self, shallow agreements on sponsored items is bad.
Our initial schedule was broken and meaningless. Just like reality in any regular projects. We changed the plan and managed to add some kind of debriefing. A few teams had gotten very far while others had just started. We realized that the time needed to get started was long. Something to consider for coming test labs. The teams debriefed and expressed what they had found. It was not a perfect first day of the test lab.
The next day, directly after lunch, some of the conference participants joined us in the test lab to run a few sessions, exploring XBMC. It was nice to test together, sharing techniques and looking at issues found. At 20.00 we started the test lab for the second night. This time people were a bit more prepared, more were on time and the focus was great. James Lyndsay had the great idea of creating certificates, somewhat fake ones, that we had posted on a wall to be handed out both of the nights. Our idea was that these should hint the participants on what was valued in the test lab, thus emphasis on diversity, creativity, persistence and so on. Participants were to hand the certificates out to others who they thought were worthy of them. James elegantly created most of them, I believe I can only created the one called Best dressed tester.
The participants in the test lab is used to being a bit stressed with tight deadlines and short scenarios/events. But this time, we went around to the teams at a bit after 8.00 and told them we are holding a debrief 9.40, letting them dig deep and focus. The room was fully of energy and focus. Michael Bolton worked on a mindmap and exploration of Mumble. Pradeep did the same, but investigated the black boxes created by Altom, based on Flash programs from James. I was surprised that so many wanted to focus on the Mumble application, but I guess I was a bit biased with my focus on XBMC.
Close to 10.00 we started to debrief and each time presented what they had found. Many of the participants were not used to working collaboratively with planning, testing and reporting. Some said that had learned a lot by watching testers present as well as trying out new tools in their collaboration. When everyone was done, it seemed like a great success. One of the reason for success were the participants sharing techniques, tools, ideas and showing how they tested. Those who participated could at the end of the conference say that, “Yes, I was at Let’s Test and I tested!”, which I think is a great.
Summing it up, I would like to thank Compare Testlab again for helping out in the lab, James Lyndsay for being such a great partner and to all who joined to participate in the test lab during the conference. I am also thankful to get lots of experience with shallow agreements even if it brought me lots of trouble.