When is a system small, non-complex or unprioritzed enough not to be tested? If there is a test organisation working on the bigger system that will be released to customer, what happens to the other smaller systems then? Is it so that they are almost always left untested? I usually identify these as applications that are created by one person, they can be used in critical parts of the product development, but are not viewed at as a product, sub-system or application that needs to be tested.

During mid 90’s I worked at a small localization company. At that time I was involved in a project where we were preparing a major translation of swedish material that was going to be translated to 20 languages. In order to succeed with such a translation it was important that we first translated the material into english and then went from there to the other languages. I did a little script to fix the glossaries for the various languages. It was just a small script to prepare for translation to the various languages. At that time we had a few testers but they were either testing commercial applications or testing for our customers, so there were no resources left for my stuff to be tested. Naturally there were loads of bugs in my code. I accidently made an offset in the script for each row, so each translated language used my glossaries with an offset of the translation. The proofreaders noticed the error eventually, but it had cost the company a few millions by then.

During another project we had a problem working with our customers files. A customer required us to use a specific program that used one working folder which contained close 100000 RTF-documents. Each time a translator checked the folder to find a specific document it took him a few hours just to see the file. So, when we wanted to do many changes to many files at the same time we calculated this to take 3 years time in order to implement our changes. So, we did a little perl script to do these changes we want by searching and replacing directly in the RTF-format (very dangerous way). First we did just one little change and when we saw that this worked well, we added more. Eventually we had a quite long list of changes being done to these 100000 files. We probably saved a few hundred years of time and budget, but eventually there were bugs in the script. One of these bugs where that I thought I had fixed issues with too many spaces and replaced them with just one space, but RTF sometimes needed two or more spaces. The result was that whole sections of text disappeared totally from being shown. This was just a small tool.

At most companies there are many such small systems floating around in the organisation. Shall the test organisation take a step toward testing those? If they are so small might it be so that testing also takes little effort? How about automation tests or production tests? Are they prioritized to be tested or do they fall into the same unimportant category?

If we see testing as a service, I think it is important that the test organisation involve themselves in all development, even if it is a few minutes time. Developers also need to involve testers, to give them a chance to do their job. Naturally it will be impossible to do everything, but in some cases it only takes such a short time to find a few bugs that it is worth it.

As a user of Facebook I feel really helpless when nothing works as it should (as was the case with the latest GUI-update). Posts were stochastically shown in the feed and a lot of errors occurred in various situations. A lot (all?) of my friends on Facebook experienced the same problems.

When there are lots of bugs on flight booking sites I get so frustrated and angry because I cannot complete my task. E.g., the booking system for Ryanair (at least two years ago when I last used their booking system and never will use again).

But what can I do as a frustrated enduser?

The problem nowadays is that you as a user and consumer don’t know what to do with your complaints. The companies shows no respect as long as the users are hooked on the application and needs the solution to problem that the application provides. But they do not care to fix the bugs that might ranging from annoying to critical.
So the loser is you! And it becomes a rage against the machine because of the absence of real people to talk to. Or some people might think that they did something wrong when it really was lousy software that caused the problems.

But in fact all other users of these applications experience similar problems. So you as an enduser is not alone! This is something that hits many people on their own but who lacks a community i.e., if we were employees in a company using the software we would have protested.

What should we do!?

Should we create a site similar to http://www.badsoftware.com/ (Kaner & Pels) or http://www.badsoftware.com/alienwaresucks/ (Kaner) in order to channel some of the frustration and gather it into some powerful criticism?

Or are there other channels that we could utilize?

Do you have any ideas?

As a Swede, the winter Olympics are fun to watch. Most sports aren’t spread across the globe, so we have chances for medals.
One of the most exciting events are the Biathlon, and perhaps it was a reaction to disappointing results for the Swedish ladies, but I got really upset at the software:

The numbers for the time are too difficult to read on a non-wide-screen TV. I have a 28 inches box from 2000, and get tired from trying to distinguish 32 from 39.
So an estimate is that more than half of the viewers have trouble seeing what time the athletes have at certain moments.
They also have graphics for shots not fired, hits, misses; and the coloring choice for not shot and misses are too close, and I don’t even have a color-blindness (I think…)
The main graphics from the software is too cluttered, which gets apparent when an athlete is shown in a corner with only parts of the graphics, and everything is much more clear (although the space is smaller.)

And everything is connected, so the software doesn’t only have Usability problems, there are functional issues as well:
When athletes are shooting, their target plate can be displayed multiple times, e.g. at one occasion it was displayed that the Norwegian favorite Björndalen shot at two targets, at one starting from the left, and on the other from the right. When he had finished, it was Teela that had two targets…
On a replay, Helena Jonsson’s time was 0.0.0.0 and matched against a leading time of unknown source.

There has also been missing features, in the latest events they have added an indicator of the wind, and also you can see the time difference between the athlete and the leader (or the second place if it is the best time.) When the inevitable happened, that the time was exactly the same as the leader, the graphics displayed -11.0…

I have no idea how this software has been developed, if they have done any sort of testing, but the addition of features and apparent bugs indicate a rushed project without input from people that will watch the event on “real” TV sets.

But since Björn Ferry won gold, I am not upset anymore; rather this software can be seen as a nice opportunity to practice your ability to see anomalies in real time software.

Session based testing is most often discussed in combination with exploratory testing. The idea is to make exploratory testing more structured by using it, as James Bach expresses it. I like the whole concept about testing in sessions, thus breaking the day into chunks of work. Considering that you know that there is an infinite set of tests possible, that it is easy to get side-tracked by meetings and other activities, you can use these sessions as a way to gain focus on testing and notify others that this time is holy. You can take the whole package with what this means, but you could also start with working in sessions. Further down the road it is possible to take the transition into using the whole spectra of Session Based Testing Management (SBTM).

Based on my own observations… it seems like it is easy for observers to understand the benefits of using SBTM. The concept of focus, defocus and debrief are quite self-explaining. The initial response is positive from personel utterly new to testing. Some comments I’ve heard was that it feels like you get things done and that you can easier move on to the next task at hand. I also notice that the test group seem to collaborate more, enjoy themselves in the testing tasks and gain confidence as testers. The hard part is the continuous documentation, known as a charter, created in order to make each session accountable. I see this as a natural step in becoming a good tester. If you cannot explain what you have been doing, cannot show or present some kind of results or observations, I would question what you really had been doing.

If you are in an organisation (probably a big one) where process changes takes time and might need a long line of approval, this might be something worth trying out. It is possible to implement directly into the test team. What you put in these sessions is up to you. The mission for each test session can be many different things, such as a group of test cases or a few test ideas or risks worth exploring. Using SBTM does not have to replace the way you work today, but complement it by adding a new and, as I see it, improved way of handling the exploratory part of testing.

I grew up in a small “town” in Värmland. Outside the village, most people live in isolated houses/farms on the countryside or in the woods.
If you’d ask one of these persons what quality is, they would answer:

dä ä väl att fôlk töcker att dä ä bra
(guess it’s that people like it)

This is actually quite a nice summary, well in line with Jerry Weinbergs “quality is value to some person”; but it doesn’t help us a lot as software testers (except clearly pointing at quality being something subjective.)
I read Marlena Compton’s blog post on quality and art (http://marlenacompton.com/?p=605) and realized that quality, like (good) art, isn’t so hard to distinguish when you see it, at least not if it’s quality for yourself. So here’s a try at a slightly more fruitful definition for testers:

quality is value to me, people I know, or people I don’t know

If I am a customer, or am using the software with a customer’s understanding and needs, it is not difficult to understand what is important and needs to be fixed.
If I know a lot about the customers and the area where the product is operating, it is also not so difficult to evaluate if the software is “good”.
If I don’t know the users, I can only take guesses at what quality could be for the product.
So as testers we should try to get more “Me” people, maybe by using the product internally (eat our own dog food, drink our own champagne/pommac), and we should try to move “Don’t Know” to “Know” by learning, using Personas, learning, visiting customers, learning etc.

But when reading TASSQ magazine from 2006 (http://www.tassq.org/quarterly/docs/TASSQuarterly0604B.pdf) there were some definitions that said that we don’t have to be so subjective in the quality definition, e.g. if it is possible to say that a car with certain attributes is of high quality, why can’t we do it for software.
This leads me to another try:

quality is more than the subjective sum of relevant quality attributes like
capability, trustworthiness, usability, security, performance, compatibility, entertaining…

You can use many, many different quality attributes, and the importance will differ for products and types of users, but by thinking about the right ones, you will get a hint on what would be better or worse quality.
One caution: quality is “more” than the sum of the attributes, there is something unknown and magical that some software has, and some don’t.

All in all; I don’t know what software quality is, but I know that it is very important and something we should continuously think about.
We won’t come up with something universally true, but it can be useful.

When looking for personel in general it is common that we want passionate people who love their work. Most passionate testers that I read about are usually part of the context-driven movement. Can there be testers out there that are really passionate about how they work in the heavy scripted test environment, where someone else does the thinking, writes the script and then hands it over for execution? What personal goals do you have when doing these tasks? Is your main focus in moving out of that zone so you also can do the thinking, write the test script and pass it on?

In the ET heavy environment I tend to see fewer roles and career paths, is this true? There are test leads, test managers and testers. Everyone want to test since that is their passion, no? You have fewer administrators and more doers.

In the scripted test environment I see more different roles and career paths. There are experts in certain domains, writing test cases. There are many test leads who spend their time trying to lead what is to be tested and exactly how. There are test managers allocating the very few testers in executing these tests. There are more administrators and less doers. A few times I’ve seen almost 90% administrators and 10% testers. With administrator I mean someone who has little focus on actually doing any tests themselves, and in some cases do not want to. One common management idea is that anyone can assist with executing a test script. This means that anyone can be a testers, thus you are not special in being a tester and you are easily replaced. There is no skill in being a tester since you just follow that script, no?

Are you then passionate about being a script tester?

Analogies are helpful, not because they come with truths, but because they can help you highlight and think in different ways about the phenomena you are comparing with.
I think you can pick any subject you know a lot about, and after some thinking, interesting things will emerge.

The important moments
If two chess players are at about the same strength, the winner often is the player that realizes at which stage it is necessary to re-think the strategy very carefully. Some moves are more important than others.
For software projects that don’t go perfectly well, and when unknown things happen, the better activities might be straightforward, or require a lot of consideration and creativity.

Technique
There are many typical methods that every good chess player must know about (forks, rook endings etc.) in order to see opportunities and apply them in the right situations.
Software projects differ much, much more than chess games, but there are many available quicktests, tricks and test design techniques that you can make good use of, if you know them well.

Theory
In chess there has been an enormous amount of analysis of different opening moves, and a player has a great advantage if she knows a lot about how to start games, and the typical positions and strategies that are likely.
Since each project have a new starting position, we can’t have this in testing.

Understand what is important
In chess there are some key elements the player needs to think about (material, development, centre, king’s safety, pawn structure etc.), but in each game these different aspects have different importance; it doesn’t matter if you have a great advantage on the Queen’s side if your opponent is mating in three.
It is the same in testing, we might know beforehand which areas and attributes that matter, but since testing can’t be complete, and unknown things always happen, we need to adjust and focus on all those things that are most important.

Time trouble
If you spend too much time on your moves, you will end up in time trouble, and once there, it is a much bigger risk of making big mistakes.
We can get in the same situation in testing (e.g. a fixed release date, even though everything else has been pushed), with the main difference that the test team have little chance of avoiding this by their own means.

“it is better with a bad plan, than no plan at all”
When you learn chess, you are often told that you must have a plan, that having no plan is a bigger mistake. Later, I have realized that a bad plan probably is worse, but by always creating a plan, you get better at it.
So we have the same in testing: it is essential to have a plan, and a bad plan is better than no plan from a learning perspective.

Practicing
There are many ways to learn chess, but a key element is to play a lot; and I think it is the same for testing.

Analysis of games
After a competition game where you have played for a couple of hours, it is common that the two opponents sit together and go through the game, move by move. They talk about their thinking, and examine what would have happened if better moves had been chosen.
The exact same thing is difficult to do for testing, but detailed retrospectives of important decisions or bugs can be a great learning exercise, and we should do more of this. Maybe directly after a pair session?

History
When learning chess you study the history, look at classic games that you learn from and get inspired by.
We don’t have this in software testing, and it is a pity.

Diversity
The people playing chess are of all different types, and like in testing with all types of backgrounds. Maybe the concentration of peculiar people is higher, both in chess and testing; this is a good thing.

Fun
You learn more when enjoying yourself, both in chess and in software testing.

When making comparisons it can be fair to list the most important differences:
* Chess is a game, testing isn’t.
* Team work and collaboration is very different.
* Chess has a defined play area, and a specific set of pieces; and this is certainly not the case for any two software projects.

This is an idea you see and hear now and then. It comes in different shapes, ranging from testers needing to have an independent manager, to testers being best if physically separated from developers, or even outsourced, or crowdsourced.

Cem Kaner writes in The Ongoing Revolution in Software Testing that this notion primarily is a “fear of bias”, and he is right, as always.
The rationale for independent testing boils downs to testing not being influenced by the development. Of course the influence can sometimes be negative, but the positive influence is more common, and more important.

There are three important reasons why I wouldn’t want to work separated from development:
* More and better information (both ways)
* More ownership and motivation
* Working together with different knowledge and focus gives a holistic view, and a better end result.

But since this separation can take so many forms, there is a need for some clarifications.
* If the testing team belong to a separate group hierarchically, but in all positive ways can influence and be influenced by other groups, I see no problem with it.
* If outsourcing includes developers, testers, documenters, and maybe even product management, I see no problem.
* it is OK if parts of the test effort are performed elsewhere, in order to get different views and approaches; Beta testing is an excellent example.
* if the ambition isn’t higher than doing exactly what is specified in detailed requirements, separation might even make that easier.
* Regardless of setup, if there is a mentality of minding your own business only, I don’t think it is a good setup.

I’m not sure if it is the physical or the mental separation that undermines the ability to get involved in each other’s work, which I see as a good thing, not because of lack of trust, but because we wanna make sure that the end result is a great product.

I’ve recently finished James Bach’s book Secrets of a Buccaneer Scholar. I liked it, but I don’t agree with all of it. As a tester, I feel that it inspires me and gives me new ideas in my way of thinking and how I perceive learning, especially self-education. I fully agree with James on that you should follow your passion. If you are able to assist those around you to that end, it will make you grow even more. In march James will be in Sweden and hold a series of courses, one of them is Self-Education for testers. It think it would be very fun and educational to attend, I will see if I can make time.

One thing that I consider after reading the book is how I can inspire my daughter to learn, test new things and to follow her passion. When she receives a new toy, I want her to explore how it works. For instance, she got a little toy puppy that execute somersaults. It had a lot of mechanic inside it and sounded like it was very fragile, if you actually played with it. The purpose of the toy was probably just to watch it. I dislike such toys. I asked my daughter how she thought it worked. She did a somewhat exploratory, destructive test by enabling the puppy to do the somersault, then directly afterwards hugged it tightly. There was a popping sound in the mechanic as the puppy tried to execute the somersault, while being held secure. After that test it was not possible for the puppy to somersault, rather it performed a half one and landed on its nose. I applauded my daughters destructive sense of testing of the toy. Translating that into testing terms, she prioritized which test to start with and considered what was the biggest chance a user would do then executed that. One test to render the system useless. Wonderful!

I think we have a lot to learn (or perhaps relearn) from our children.

Now and then I read that testers don’t know enough about Usability, that there is a need for a Performance Testing expert, that a Security consultant should be called in, or that a master of the used technology would make Installation and Compatibility testing possible.
This might be true in the general case, but there are many testers that are working with the same product suite for years, and over time you become an expert of most aspects of quality for your specific product.

Let’s look at some examples:

One sub-category of Usability is Operability; that an experienced user can perform functions in a fast and efficient way. As a system tester, you have done most operations many times, and you know what you can expect from features in the sense of operability. You can point at places where for instance the ability to delete multiple items with a few clicks would make a difference.
Learnability is another sub-category of Usability. You only learn the basics once, but maybe you have heard customer stories of confusing things, or you could let a new member of the test team think in this direction.
Regarding Accessibility it doesn’t cost too much to use High DPI, speakers turned on, Code Blind or Magnifier sometimes.

Large-scale Performance might require an expensive tool, but I bet you’re doing some simulations without them; maybe just by telling the whole development team to hit the same server at the same time.
There is also a small-scale, low-level Performance aspect that shouldn’t be understimated. If a dialog takes more than a second to display, it might be something that put the user’s confidence in doubt.
If you have tested your product for quite some time, you will immediately notice when something takes just a bit longer than it could take. There of course might be valid reasons for this, but talking to the developer about it might be beneficial to both the producers and consumers of your software.

Security testing seems more difficult, but as a product tester you know at which moments authentication takes place, you know if there are passwords stored, and that they should be encryted; you might not know how to exploit a crash, but you are an expert at provoking the crashes.

For Hardware/OS/Application Compatibility testing you will become more of an expert the longer you work with them, at least if you have the curiousity to learn more things when you get the chance.
You might not know a lot about how the iPhone works, but you know all the details that can be used for interacting with your web site.

Sometimes I also read the extreme that testers shouldn’t bother with Usability/Security/Performance testing, which to me seems like an incredible waste of knowledge and resources.
When testing functionality manually, you can look at quality attributes at the same time, and get a lot of coverage for free.

I’m not saying true specialists aren’t needed, but I’m saying that there are a lot of expertise in your building that at least can be used complementary.
If you are in a situation where you know a lot of these things, but aren’t allowed/encouraged to test these things, I think you should try to convince your managers of a better and more fun way to test your product.