Vancouver 2010 Biathlon Software Rikard Edgren 5 Comments

As a Swede, the winter Olympics are fun to watch. Most sports aren’t spread across the globe, so we have chances for medals.
One of the most exciting events are the Biathlon, and perhaps it was a reaction to disappointing results for the Swedish ladies, but I got really upset at the software:

The numbers for the time are too difficult to read on a non-wide-screen TV. I have a 28 inches box from 2000, and get tired from trying to distinguish 32 from 39.
So an estimate is that more than half of the viewers have trouble seeing what time the athletes have at certain moments.
They also have graphics for shots not fired, hits, misses; and the coloring choice for not shot and misses are too close, and I don’t even have a color-blindness (I think…)
The main graphics from the software is too cluttered, which gets apparent when an athlete is shown in a corner with only parts of the graphics, and everything is much more clear (although the space is smaller.)

And everything is connected, so the software doesn’t only have Usability problems, there are functional issues as well:
When athletes are shooting, their target plate can be displayed multiple times, e.g. at one occasion it was displayed that the Norwegian favorite Björndalen shot at two targets, at one starting from the left, and on the other from the right. When he had finished, it was Teela that had two targets…
On a replay, Helena Jonsson’s time was and matched against a leading time of unknown source.

There has also been missing features, in the latest events they have added an indicator of the wind, and also you can see the time difference between the athlete and the leader (or the second place if it is the best time.) When the inevitable happened, that the time was exactly the same as the leader, the graphics displayed -11.0…

I have no idea how this software has been developed, if they have done any sort of testing, but the addition of features and apparent bugs indicate a rushed project without input from people that will watch the event on “real” TV sets.

But since Björn Ferry won gold, I am not upset anymore; rather this software can be seen as a nice opportunity to practice your ability to see anomalies in real time software.

Session-based testing as a foundation for test activities Martin Jansson No Comments

Session based testing is most often discussed in combination with exploratory testing. The idea is to make exploratory testing more structured by using it, as James Bach expresses it. I like the whole concept about testing in sessions, thus breaking the day into chunks of work. Considering that you know that there is an infinite set of tests possible, that it is easy to get side-tracked by meetings and other activities, you can use these sessions as a way to gain focus on testing and notify others that this time is holy. You can take the whole package with what this means, but you could also start with working in sessions. Further down the road it is possible to take the transition into using the whole spectra of Session Based Testing Management (SBTM).

Based on my own observations… it seems like it is easy for observers to understand the benefits of using SBTM. The concept of focus, defocus and debrief are quite self-explaining. The initial response is positive from personel utterly new to testing. Some comments I’ve heard was that it feels like you get things done and that you can easier move on to the next task at hand. I also notice that the test group seem to collaborate more, enjoy themselves in the testing tasks and gain confidence as testers. The hard part is the continuous documentation, known as a charter, created in order to make each session accountable. I see this as a natural step in becoming a good tester. If you cannot explain what you have been doing, cannot show or present some kind of results or observations, I would question what you really had been doing.

If you are in an organisation (probably a big one) where process changes takes time and might need a long line of approval, this might be something worth trying out. It is possible to implement directly into the test team. What you put in these sessions is up to you. The mission for each test session can be many different things, such as a group of test cases or a few test ideas or risks worth exploring. Using SBTM does not have to replace the way you work today, but complement it by adding a new and, as I see it, improved way of handling the exploratory part of testing.

More Definitions of Quality Rikard Edgren 12 Comments

I grew up in a small “town” in Värmland. Outside the village, most people live in isolated houses/farms on the countryside or in the woods.
If you’d ask one of these persons what quality is, they would answer:

dä ä väl att fôlk töcker att dä ä bra
(guess it’s that people like it)

This is actually quite a nice summary, well in line with Jerry Weinbergs “quality is value to some person”; but it doesn’t help us a lot as software testers (except clearly pointing at quality being something subjective.)
I read Marlena Compton’s blog post on quality and art ( and realized that quality, like (good) art, isn’t so hard to distinguish when you see it, at least not if it’s quality for yourself. So here’s a try at a slightly more fruitful definition for testers:

quality is value to me, people I know, or people I don’t know

If I am a customer, or am using the software with a customer’s understanding and needs, it is not difficult to understand what is important and needs to be fixed.
If I know a lot about the customers and the area where the product is operating, it is also not so difficult to evaluate if the software is “good”.
If I don’t know the users, I can only take guesses at what quality could be for the product.
So as testers we should try to get more “Me” people, maybe by using the product internally (eat our own dog food, drink our own champagne/pommac), and we should try to move “Don’t Know” to “Know” by learning, using Personas, learning, visiting customers, learning etc.

But when reading TASSQ magazine from 2006 ( there were some definitions that said that we don’t have to be so subjective in the quality definition, e.g. if it is possible to say that a car with certain attributes is of high quality, why can’t we do it for software.
This leads me to another try:

quality is more than the subjective sum of relevant quality attributes like
capability, trustworthiness, usability, security, performance, compatibility, entertaining…

You can use many, many different quality attributes, and the importance will differ for products and types of users, but by thinking about the right ones, you will get a hint on what would be better or worse quality.
One caution: quality is “more” than the sum of the attributes, there is something unknown and magical that some software has, and some don’t.

All in all; I don’t know what software quality is, but I know that it is very important and something we should continuously think about.
We won’t come up with something universally true, but it can be useful.

Are there any passionate script testers? Martin Jansson 10 Comments

When looking for personel in general it is common that we want passionate people who love their work. Most passionate testers that I read about are usually part of the context-driven movement. Can there be testers out there that are really passionate about how they work in the heavy scripted test environment, where someone else does the thinking, writes the script and then hands it over for execution? What personal goals do you have when doing these tasks? Is your main focus in moving out of that zone so you also can do the thinking, write the test script and pass it on?

In the ET heavy environment I tend to see fewer roles and career paths, is this true? There are test leads, test managers and testers. Everyone want to test since that is their passion, no? You have fewer administrators and more doers.

In the scripted test environment I see more different roles and career paths. There are experts in certain domains, writing test cases. There are many test leads who spend their time trying to lead what is to be tested and exactly how. There are test managers allocating the very few testers in executing these tests. There are more administrators and less doers. A few times I’ve seen almost 90% administrators and 10% testers. With administrator I mean someone who has little focus on actually doing any tests themselves, and in some cases do not want to. One common management idea is that anyone can assist with executing a test script. This means that anyone can be a testers, thus you are not special in being a tester and you are easily replaced. There is no skill in being a tester since you just follow that script, no?

Are you then passionate about being a script tester?

Chess & Testing Rikard Edgren 6 Comments

Analogies are helpful, not because they come with truths, but because they can help you highlight and think in different ways about the phenomena you are comparing with.
I think you can pick any subject you know a lot about, and after some thinking, interesting things will emerge.

The important moments
If two chess players are at about the same strength, the winner often is the player that realizes at which stage it is necessary to re-think the strategy very carefully. Some moves are more important than others.
For software projects that don’t go perfectly well, and when unknown things happen, the better activities might be straightforward, or require a lot of consideration and creativity.

There are many typical methods that every good chess player must know about (forks, rook endings etc.) in order to see opportunities and apply them in the right situations.
Software projects differ much, much more than chess games, but there are many available quicktests, tricks and test design techniques that you can make good use of, if you know them well.

In chess there has been an enormous amount of analysis of different opening moves, and a player has a great advantage if she knows a lot about how to start games, and the typical positions and strategies that are likely.
Since each project have a new starting position, we can’t have this in testing.

Understand what is important
In chess there are some key elements the player needs to think about (material, development, centre, king’s safety, pawn structure etc.), but in each game these different aspects have different importance; it doesn’t matter if you have a great advantage on the Queen’s side if your opponent is mating in three.
It is the same in testing, we might know beforehand which areas and attributes that matter, but since testing can’t be complete, and unknown things always happen, we need to adjust and focus on all those things that are most important.

Time trouble
If you spend too much time on your moves, you will end up in time trouble, and once there, it is a much bigger risk of making big mistakes.
We can get in the same situation in testing (e.g. a fixed release date, even though everything else has been pushed), with the main difference that the test team have little chance of avoiding this by their own means.

“it is better with a bad plan, than no plan at all”
When you learn chess, you are often told that you must have a plan, that having no plan is a bigger mistake. Later, I have realized that a bad plan probably is worse, but by always creating a plan, you get better at it.
So we have the same in testing: it is essential to have a plan, and a bad plan is better than no plan from a learning perspective.

There are many ways to learn chess, but a key element is to play a lot; and I think it is the same for testing.

Analysis of games
After a competition game where you have played for a couple of hours, it is common that the two opponents sit together and go through the game, move by move. They talk about their thinking, and examine what would have happened if better moves had been chosen.
The exact same thing is difficult to do for testing, but detailed retrospectives of important decisions or bugs can be a great learning exercise, and we should do more of this. Maybe directly after a pair session?

When learning chess you study the history, look at classic games that you learn from and get inspired by.
We don’t have this in software testing, and it is a pity.

The people playing chess are of all different types, and like in testing with all types of backgrounds. Maybe the concentration of peculiar people is higher, both in chess and testing; this is a good thing.

You learn more when enjoying yourself, both in chess and in software testing.

When making comparisons it can be fair to list the most important differences:
* Chess is a game, testing isn’t.
* Team work and collaboration is very different.
* Chess has a defined play area, and a specific set of pieces; and this is certainly not the case for any two software projects.

Testing Clichés Part II – Testing should be separate from development Rikard Edgren 5 Comments

This is an idea you see and hear now and then. It comes in different shapes, ranging from testers needing to have an independent manager, to testers being best if physically separated from developers, or even outsourced, or crowdsourced.

Cem Kaner writes in The Ongoing Revolution in Software Testing that this notion primarily is a “fear of bias”, and he is right, as always.
The rationale for independent testing boils downs to testing not being influenced by the development. Of course the influence can sometimes be negative, but the positive influence is more common, and more important.

There are three important reasons why I wouldn’t want to work separated from development:
* More and better information (both ways)
* More ownership and motivation
* Working together with different knowledge and focus gives a holistic view, and a better end result.

But since this separation can take so many forms, there is a need for some clarifications.
* If the testing team belong to a separate group hierarchically, but in all positive ways can influence and be influenced by other groups, I see no problem with it.
* If outsourcing includes developers, testers, documenters, and maybe even product management, I see no problem.
* it is OK if parts of the test effort are performed elsewhere, in order to get different views and approaches; Beta testing is an excellent example.
* if the ambition isn’t higher than doing exactly what is specified in detailed requirements, separation might even make that easier.
* Regardless of setup, if there is a mentality of minding your own business only, I don’t think it is a good setup.

I’m not sure if it is the physical or the mental separation that undermines the ability to get involved in each other’s work, which I see as a good thing, not because of lack of trust, but because we wanna make sure that the end result is a great product.

Passion, self-education and testing Martin Jansson 4 Comments

I’ve recently finished James Bach’s book Secrets of a Buccaneer Scholar. I liked it, but I don’t agree with all of it. As a tester, I feel that it inspires me and gives me new ideas in my way of thinking and how I perceive learning, especially self-education. I fully agree with James on that you should follow your passion. If you are able to assist those around you to that end, it will make you grow even more. In march James will be in Sweden and hold a series of courses, one of them is Self-Education for testers. It think it would be very fun and educational to attend, I will see if I can make time.

One thing that I consider after reading the book is how I can inspire my daughter to learn, test new things and to follow her passion. When she receives a new toy, I want her to explore how it works. For instance, she got a little toy puppy that execute somersaults. It had a lot of mechanic inside it and sounded like it was very fragile, if you actually played with it. The purpose of the toy was probably just to watch it. I dislike such toys. I asked my daughter how she thought it worked. She did a somewhat exploratory, destructive test by enabling the puppy to do the somersault, then directly afterwards hugged it tightly. There was a popping sound in the mechanic as the puppy tried to execute the somersault, while being held secure. After that test it was not possible for the puppy to somersault, rather it performed a half one and landed on its nose. I applauded my daughters destructive sense of testing of the toy. Translating that into testing terms, she prioritized which test to start with and considered what was the biggest chance a user would do then executed that. One test to render the system useless. Wonderful!

I think we have a lot to learn (or perhaps relearn) from our children.

You might be an expert at non-functional testing Rikard Edgren 2 Comments

Now and then I read that testers don’t know enough about Usability, that there is a need for a Performance Testing expert, that a Security consultant should be called in, or that a master of the used technology would make Installation and Compatibility testing possible.
This might be true in the general case, but there are many testers that are working with the same product suite for years, and over time you become an expert of most aspects of quality for your specific product.

Let’s look at some examples:

One sub-category of Usability is Operability; that an experienced user can perform functions in a fast and efficient way. As a system tester, you have done most operations many times, and you know what you can expect from features in the sense of operability. You can point at places where for instance the ability to delete multiple items with a few clicks would make a difference.
Learnability is another sub-category of Usability. You only learn the basics once, but maybe you have heard customer stories of confusing things, or you could let a new member of the test team think in this direction.
Regarding Accessibility it doesn’t cost too much to use High DPI, speakers turned on, Code Blind or Magnifier sometimes.

Large-scale Performance might require an expensive tool, but I bet you’re doing some simulations without them; maybe just by telling the whole development team to hit the same server at the same time.
There is also a small-scale, low-level Performance aspect that shouldn’t be understimated. If a dialog takes more than a second to display, it might be something that put the user’s confidence in doubt.
If you have tested your product for quite some time, you will immediately notice when something takes just a bit longer than it could take. There of course might be valid reasons for this, but talking to the developer about it might be beneficial to both the producers and consumers of your software.

Security testing seems more difficult, but as a product tester you know at which moments authentication takes place, you know if there are passwords stored, and that they should be encryted; you might not know how to exploit a crash, but you are an expert at provoking the crashes.

For Hardware/OS/Application Compatibility testing you will become more of an expert the longer you work with them, at least if you have the curiousity to learn more things when you get the chance.
You might not know a lot about how the iPhone works, but you know all the details that can be used for interacting with your web site.

Sometimes I also read the extreme that testers shouldn’t bother with Usability/Security/Performance testing, which to me seems like an incredible waste of knowledge and resources.
When testing functionality manually, you can look at quality attributes at the same time, and get a lot of coverage for free.

I’m not saying true specialists aren’t needed, but I’m saying that there are a lot of expertise in your building that at least can be used complementary.
If you are in a situation where you know a lot of these things, but aren’t allowed/encouraged to test these things, I think you should try to convince your managers of a better and more fun way to test your product.

Questions that testing constantly help answering Rikard Edgren 1 Comment

I have been thinking about qualitative research lately, and wondered what the question(s) would look like if testing was seen as a research project.

The testing effort has many positive effects, but one common and important is to provide information about the product, so a good release decision can be made. We cannot prove that the software is good, but we can try our very best to show important ways it can fail.
But at the same time, it is too late to come with important information just before the release is to be made; the information should be provided in a timely manner.
So in a paradoxal way, we try to destroy the product we love, but at the same time give the information to the project, so they can make the testing mission fail.

Testing try to answer “No” to the “Can we ship?” question – many times in many ways – but gives the project the best chances to resolve all issues as early as possible.

Another important question is “How can the product be better?”, and information about this is also being provided all the time. Not too late, but also not too early, in times when we know too little, and the cost is too high to reach the vital information.

Of course, there are many other sources and considerations, but testing often has the best view of the system as a whole, and the best view of some details from a customer perspective.
Testing can’t be complete, but the answers are valid when the testing can be considered saturated, when too much effort is needed to get new information that aren’t too important.

Are there other questions that are more important?
Or are we providing answers without questions?

Kaner’s Gold Mine Martin Jansson 2 Comments

Cem Kaner has updated his set of publications. I’ve been reading his well written articles over the last ten years.

Have a nice time digging in!