thoughts from the test eye

Over the years I have noticed that System Testing have had a special meaning at every place I have been at; and it has even meant different things for people on the same place. I.e. System Testing is depending on the context; and it is fuzzy because we are dealing with arbitrary and/or general systems.

“System testing of software or hardware is testing conducted on a complete, integrated system…” but the boundaries on what the complete and integrated system includes may vary a lot if you look at it from an outside perspective as well as if you look at it from within.

Some examples: When developing a cell phone, where does the “system” begin and where does it end? Does it begin with the cell phone OS or the phone (hardware + OS + software)? Do we need to include any network? How large network? What if you are a third party and developing the phone OS (that could be run standalone in a simulator)? What if you develop an application for the phone (or even different kind of phones)?

So when is a system “complete and integrated”?

I guess that it is very hard to define this because the boundaries of system testing are elastic, and this is true since the boundaries of an arbitrary system are hard to define. Still, we use the term “system testing” daily.

So what is my point really?

I have often found interesting things just by thinking on what system I am working with. I have found out that some things are really outside the boundaries that I first believed were in; and perhaps more useful has been to find things that really should be included in my system and the system testing. And the boundaries might stretch the further you get in a project; what was a complete and fully integrated system during the first half of the project might just be a sub-part at the end of the project.

So my advice is that you take a few minutes now and then in order to seriously ponder the boundaries of Your system. It will affect the type of testing You choose to focus on; and it will be a good aid in order to get a proper and valid test coverage.

In many years I have loved most of what is written by the people behind the context-driven school of testing.
But I have also felt that there is something that isn’t a perfect match.

For a time I thought it was because I saw a few different tries to push people to different schools – which I had ethical problems with – but I realize it is possible to belong to one school, but not push opinions and school membership onto others.

I also had the idea that the school had stopped focusing on people (who aren’t mentioned in What is context-driven testing?), but I realize that was just one blog post, people are still the most important part of the context, see http://www.context-driven-testing.com/

I came a little bit closer when I disliked CAST 2009 theme Serving Our Stakeholders, feeling that testers were diminished and put in a hierarchy of some sort (I would rather focus on the whole team with joint ownership.) But I have realized that serving in this context means giving, not doing what somebody else has ordered.

Finally I have come to the conclusion that the reason why I’m not a context-driven tester is that there are too many contexts where I wouldn’t (wanna) work (which implies a kind of context-imperial testing.)
Examples are situations where testers aren’t allowed to step out of the scripts, or if bugs not stemming from requirements are ignored, or situations where you don’t interact in any ways with other parts of the development, and other departments, or where you only are allowed to be involved for a very short period of time, or where you know that no one wants to create great software, they just want to release at time.

The opposite of these examples are “very good practices for me”, and I totally embrace the exploratory style of testing, and try to use all possible methods and techniques that could be beneficial to the goal I like: world-class software.
As a tester, I don’t feel I can be context-driving, but I have the luxury of choosing which rides to join.

When is a system small, non-complex or unprioritzed enough not to be tested? If there is a test organisation working on the bigger system that will be released to customer, what happens to the other smaller systems then? Is it so that they are almost always left untested? I usually identify these as applications that are created by one person, they can be used in critical parts of the product development, but are not viewed at as a product, sub-system or application that needs to be tested.

During mid 90’s I worked at a small localization company. At that time I was involved in a project where we were preparing a major translation of swedish material that was going to be translated to 20 languages. In order to succeed with such a translation it was important that we first translated the material into english and then went from there to the other languages. I did a little script to fix the glossaries for the various languages. It was just a small script to prepare for translation to the various languages. At that time we had a few testers but they were either testing commercial applications or testing for our customers, so there were no resources left for my stuff to be tested. Naturally there were loads of bugs in my code. I accidently made an offset in the script for each row, so each translated language used my glossaries with an offset of the translation. The proofreaders noticed the error eventually, but it had cost the company a few millions by then.

During another project we had a problem working with our customers files. A customer required us to use a specific program that used one working folder which contained close 100000 RTF-documents. Each time a translator checked the folder to find a specific document it took him a few hours just to see the file. So, when we wanted to do many changes to many files at the same time we calculated this to take 3 years time in order to implement our changes. So, we did a little perl script to do these changes we want by searching and replacing directly in the RTF-format (very dangerous way). First we did just one little change and when we saw that this worked well, we added more. Eventually we had a quite long list of changes being done to these 100000 files. We probably saved a few hundred years of time and budget, but eventually there were bugs in the script. One of these bugs where that I thought I had fixed issues with too many spaces and replaced them with just one space, but RTF sometimes needed two or more spaces. The result was that whole sections of text disappeared totally from being shown. This was just a small tool.

At most companies there are many such small systems floating around in the organisation. Shall the test organisation take a step toward testing those? If they are so small might it be so that testing also takes little effort? How about automation tests or production tests? Are they prioritized to be tested or do they fall into the same unimportant category?

If we see testing as a service, I think it is important that the test organisation involve themselves in all development, even if it is a few minutes time. Developers also need to involve testers, to give them a chance to do their job. Naturally it will be impossible to do everything, but in some cases it only takes such a short time to find a few bugs that it is worth it.

As a user of Facebook I feel really helpless when nothing works as it should (as was the case with the latest GUI-update). Posts were stochastically shown in the feed and a lot of errors occurred in various situations. A lot (all?) of my friends on Facebook experienced the same problems.

When there are lots of bugs on flight booking sites I get so frustrated and angry because I cannot complete my task. E.g., the booking system for Ryanair (at least two years ago when I last used their booking system and never will use again).

But what can I do as a frustrated enduser?

The problem nowadays is that you as a user and consumer don’t know what to do with your complaints. The companies shows no respect as long as the users are hooked on the application and needs the solution to problem that the application provides. But they do not care to fix the bugs that might ranging from annoying to critical.
So the loser is you! And it becomes a rage against the machine because of the absence of real people to talk to. Or some people might think that they did something wrong when it really was lousy software that caused the problems.

But in fact all other users of these applications experience similar problems. So you as an enduser is not alone! This is something that hits many people on their own but who lacks a community i.e., if we were employees in a company using the software we would have protested.

What should we do!?

Should we create a site similar to http://www.badsoftware.com/ (Kaner & Pels) or http://www.badsoftware.com/alienwaresucks/ (Kaner) in order to channel some of the frustration and gather it into some powerful criticism?

Or are there other channels that we could utilize?

Do you have any ideas?

As a Swede, the winter Olympics are fun to watch. Most sports aren’t spread across the globe, so we have chances for medals.
One of the most exciting events are the Biathlon, and perhaps it was a reaction to disappointing results for the Swedish ladies, but I got really upset at the software:

The numbers for the time are too difficult to read on a non-wide-screen TV. I have a 28 inches box from 2000, and get tired from trying to distinguish 32 from 39.
So an estimate is that more than half of the viewers have trouble seeing what time the athletes have at certain moments.
They also have graphics for shots not fired, hits, misses; and the coloring choice for not shot and misses are too close, and I don’t even have a color-blindness (I think…)
The main graphics from the software is too cluttered, which gets apparent when an athlete is shown in a corner with only parts of the graphics, and everything is much more clear (although the space is smaller.)

And everything is connected, so the software doesn’t only have Usability problems, there are functional issues as well:
When athletes are shooting, their target plate can be displayed multiple times, e.g. at one occasion it was displayed that the Norwegian favorite Björndalen shot at two targets, at one starting from the left, and on the other from the right. When he had finished, it was Teela that had two targets…
On a replay, Helena Jonsson’s time was 0.0.0.0 and matched against a leading time of unknown source.

There has also been missing features, in the latest events they have added an indicator of the wind, and also you can see the time difference between the athlete and the leader (or the second place if it is the best time.) When the inevitable happened, that the time was exactly the same as the leader, the graphics displayed -11.0…

I have no idea how this software has been developed, if they have done any sort of testing, but the addition of features and apparent bugs indicate a rushed project without input from people that will watch the event on “real” TV sets.

But since Björn Ferry won gold, I am not upset anymore; rather this software can be seen as a nice opportunity to practice your ability to see anomalies in real time software.

Session based testing is most often discussed in combination with exploratory testing. The idea is to make exploratory testing more structured by using it, as James Bach expresses it. I like the whole concept about testing in sessions, thus breaking the day into chunks of work. Considering that you know that there is an infinite set of tests possible, that it is easy to get side-tracked by meetings and other activities, you can use these sessions as a way to gain focus on testing and notify others that this time is holy. You can take the whole package with what this means, but you could also start with working in sessions. Further down the road it is possible to take the transition into using the whole spectra of Session Based Testing Management (SBTM).

Based on my own observations… it seems like it is easy for observers to understand the benefits of using SBTM. The concept of focus, defocus and debrief are quite self-explaining. The initial response is positive from personel utterly new to testing. Some comments I’ve heard was that it feels like you get things done and that you can easier move on to the next task at hand. I also notice that the test group seem to collaborate more, enjoy themselves in the testing tasks and gain confidence as testers. The hard part is the continuous documentation, known as a charter, created in order to make each session accountable. I see this as a natural step in becoming a good tester. If you cannot explain what you have been doing, cannot show or present some kind of results or observations, I would question what you really had been doing.

If you are in an organisation (probably a big one) where process changes takes time and might need a long line of approval, this might be something worth trying out. It is possible to implement directly into the test team. What you put in these sessions is up to you. The mission for each test session can be many different things, such as a group of test cases or a few test ideas or risks worth exploring. Using SBTM does not have to replace the way you work today, but complement it by adding a new and, as I see it, improved way of handling the exploratory part of testing.

I grew up in a small “town” in Värmland. Outside the village, most people live in isolated houses/farms on the countryside or in the woods.
If you’d ask one of these persons what quality is, they would answer:

dä ä väl att fôlk töcker att dä ä bra
(guess it’s that people like it)

This is actually quite a nice summary, well in line with Jerry Weinbergs “quality is value to some person”; but it doesn’t help us a lot as software testers (except clearly pointing at quality being something subjective.)
I read Marlena Compton’s blog post on quality and art (http://marlenacompton.com/?p=605) and realized that quality, like (good) art, isn’t so hard to distinguish when you see it, at least not if it’s quality for yourself. So here’s a try at a slightly more fruitful definition for testers:

quality is value to me, people I know, or people I don’t know

If I am a customer, or am using the software with a customer’s understanding and needs, it is not difficult to understand what is important and needs to be fixed.
If I know a lot about the customers and the area where the product is operating, it is also not so difficult to evaluate if the software is “good”.
If I don’t know the users, I can only take guesses at what quality could be for the product.
So as testers we should try to get more “Me” people, maybe by using the product internally (eat our own dog food, drink our own champagne/pommac), and we should try to move “Don’t Know” to “Know” by learning, using Personas, learning, visiting customers, learning etc.

But when reading TASSQ magazine from 2006 (http://www.tassq.org/quarterly/docs/TASSQuarterly0604B.pdf) there were some definitions that said that we don’t have to be so subjective in the quality definition, e.g. if it is possible to say that a car with certain attributes is of high quality, why can’t we do it for software.
This leads me to another try:

quality is more than the subjective sum of relevant quality attributes like
capability, trustworthiness, usability, security, performance, compatibility, entertaining…

You can use many, many different quality attributes, and the importance will differ for products and types of users, but by thinking about the right ones, you will get a hint on what would be better or worse quality.
One caution: quality is “more” than the sum of the attributes, there is something unknown and magical that some software has, and some don’t.

All in all; I don’t know what software quality is, but I know that it is very important and something we should continuously think about.
We won’t come up with something universally true, but it can be useful.

When looking for personel in general it is common that we want passionate people who love their work. Most passionate testers that I read about are usually part of the context-driven movement. Can there be testers out there that are really passionate about how they work in the heavy scripted test environment, where someone else does the thinking, writes the script and then hands it over for execution? What personal goals do you have when doing these tasks? Is your main focus in moving out of that zone so you also can do the thinking, write the test script and pass it on?

In the ET heavy environment I tend to see fewer roles and career paths, is this true? There are test leads, test managers and testers. Everyone want to test since that is their passion, no? You have fewer administrators and more doers.

In the scripted test environment I see more different roles and career paths. There are experts in certain domains, writing test cases. There are many test leads who spend their time trying to lead what is to be tested and exactly how. There are test managers allocating the very few testers in executing these tests. There are more administrators and less doers. A few times I’ve seen almost 90% administrators and 10% testers. With administrator I mean someone who has little focus on actually doing any tests themselves, and in some cases do not want to. One common management idea is that anyone can assist with executing a test script. This means that anyone can be a testers, thus you are not special in being a tester and you are easily replaced. There is no skill in being a tester since you just follow that script, no?

Are you then passionate about being a script tester?

Analogies are helpful, not because they come with truths, but because they can help you highlight and think in different ways about the phenomena you are comparing with.
I think you can pick any subject you know a lot about, and after some thinking, interesting things will emerge.

The important moments
If two chess players are at about the same strength, the winner often is the player that realizes at which stage it is necessary to re-think the strategy very carefully. Some moves are more important than others.
For software projects that don’t go perfectly well, and when unknown things happen, the better activities might be straightforward, or require a lot of consideration and creativity.

Technique
There are many typical methods that every good chess player must know about (forks, rook endings etc.) in order to see opportunities and apply them in the right situations.
Software projects differ much, much more than chess games, but there are many available quicktests, tricks and test design techniques that you can make good use of, if you know them well.

Theory
In chess there has been an enormous amount of analysis of different opening moves, and a player has a great advantage if she knows a lot about how to start games, and the typical positions and strategies that are likely.
Since each project have a new starting position, we can’t have this in testing.

Understand what is important
In chess there are some key elements the player needs to think about (material, development, centre, king’s safety, pawn structure etc.), but in each game these different aspects have different importance; it doesn’t matter if you have a great advantage on the Queen’s side if your opponent is mating in three.
It is the same in testing, we might know beforehand which areas and attributes that matter, but since testing can’t be complete, and unknown things always happen, we need to adjust and focus on all those things that are most important.

Time trouble
If you spend too much time on your moves, you will end up in time trouble, and once there, it is a much bigger risk of making big mistakes.
We can get in the same situation in testing (e.g. a fixed release date, even though everything else has been pushed), with the main difference that the test team have little chance of avoiding this by their own means.

“it is better with a bad plan, than no plan at all”
When you learn chess, you are often told that you must have a plan, that having no plan is a bigger mistake. Later, I have realized that a bad plan probably is worse, but by always creating a plan, you get better at it.
So we have the same in testing: it is essential to have a plan, and a bad plan is better than no plan from a learning perspective.

Practicing
There are many ways to learn chess, but a key element is to play a lot; and I think it is the same for testing.

Analysis of games
After a competition game where you have played for a couple of hours, it is common that the two opponents sit together and go through the game, move by move. They talk about their thinking, and examine what would have happened if better moves had been chosen.
The exact same thing is difficult to do for testing, but detailed retrospectives of important decisions or bugs can be a great learning exercise, and we should do more of this. Maybe directly after a pair session?

History
When learning chess you study the history, look at classic games that you learn from and get inspired by.
We don’t have this in software testing, and it is a pity.

Diversity
The people playing chess are of all different types, and like in testing with all types of backgrounds. Maybe the concentration of peculiar people is higher, both in chess and testing; this is a good thing.

Fun
You learn more when enjoying yourself, both in chess and in software testing.

When making comparisons it can be fair to list the most important differences:
* Chess is a game, testing isn’t.
* Team work and collaboration is very different.
* Chess has a defined play area, and a specific set of pieces; and this is certainly not the case for any two software projects.

This is an idea you see and hear now and then. It comes in different shapes, ranging from testers needing to have an independent manager, to testers being best if physically separated from developers, or even outsourced, or crowdsourced.

Cem Kaner writes in The Ongoing Revolution in Software Testing that this notion primarily is a “fear of bias”, and he is right, as always.
The rationale for independent testing boils downs to testing not being influenced by the development. Of course the influence can sometimes be negative, but the positive influence is more common, and more important.

There are three important reasons why I wouldn’t want to work separated from development:
* More and better information (both ways)
* More ownership and motivation
* Working together with different knowledge and focus gives a holistic view, and a better end result.

But since this separation can take so many forms, there is a need for some clarifications.
* If the testing team belong to a separate group hierarchically, but in all positive ways can influence and be influenced by other groups, I see no problem with it.
* If outsourcing includes developers, testers, documenters, and maybe even product management, I see no problem.
* it is OK if parts of the test effort are performed elsewhere, in order to get different views and approaches; Beta testing is an excellent example.
* if the ambition isn’t higher than doing exactly what is specified in detailed requirements, separation might even make that easier.
* Regardless of setup, if there is a mentality of minding your own business only, I don’t think it is a good setup.

I’m not sure if it is the physical or the mental separation that undermines the ability to get involved in each other’s work, which I see as a good thing, not because of lack of trust, but because we wanna make sure that the end result is a great product.