Last week, me and Martin won the competition “The EuroSTAR Test Lab Apprentices”!
Read more at: http://www.eurostarconferences.com/delegates/the-test-lab-apprentice.aspx

See you in the Test Lab in Copenhagen!

Cheers,
Henrik & Martin

As a user of Facebook I feel really helpless when nothing works as it should (as was the case with the latest GUI-update). Posts were stochastically shown in the feed and a lot of errors occurred in various situations. A lot (all?) of my friends on Facebook experienced the same problems.

When there are lots of bugs on flight booking sites I get so frustrated and angry because I cannot complete my task. E.g., the booking system for Ryanair (at least two years ago when I last used their booking system and never will use again).

But what can I do as a frustrated enduser?

The problem nowadays is that you as a user and consumer don’t know what to do with your complaints. The companies shows no respect as long as the users are hooked on the application and needs the solution to problem that the application provides. But they do not care to fix the bugs that might ranging from annoying to critical.
So the loser is you! And it becomes a rage against the machine because of the absence of real people to talk to. Or some people might think that they did something wrong when it really was lousy software that caused the problems.

But in fact all other users of these applications experience similar problems. So you as an enduser is not alone! This is something that hits many people on their own but who lacks a community i.e., if we were employees in a company using the software we would have protested.

What should we do!?

Should we create a site similar to http://www.badsoftware.com/ (Kaner & Pels) or http://www.badsoftware.com/alienwaresucks/ (Kaner) in order to channel some of the frustration and gather it into some powerful criticism?

Or are there other channels that we could utilize?

Do you have any ideas?

As a Swede, the winter Olympics are fun to watch. Most sports aren’t spread across the globe, so we have chances for medals.
One of the most exciting events are the Biathlon, and perhaps it was a reaction to disappointing results for the Swedish ladies, but I got really upset at the software:

The numbers for the time are too difficult to read on a non-wide-screen TV. I have a 28 inches box from 2000, and get tired from trying to distinguish 32 from 39.
So an estimate is that more than half of the viewers have trouble seeing what time the athletes have at certain moments.
They also have graphics for shots not fired, hits, misses; and the coloring choice for not shot and misses are too close, and I don’t even have a color-blindness (I think…)
The main graphics from the software is too cluttered, which gets apparent when an athlete is shown in a corner with only parts of the graphics, and everything is much more clear (although the space is smaller.)

And everything is connected, so the software doesn’t only have Usability problems, there are functional issues as well:
When athletes are shooting, their target plate can be displayed multiple times, e.g. at one occasion it was displayed that the Norwegian favorite Björndalen shot at two targets, at one starting from the left, and on the other from the right. When he had finished, it was Teela that had two targets…
On a replay, Helena Jonsson’s time was 0.0.0.0 and matched against a leading time of unknown source.

There has also been missing features, in the latest events they have added an indicator of the wind, and also you can see the time difference between the athlete and the leader (or the second place if it is the best time.) When the inevitable happened, that the time was exactly the same as the leader, the graphics displayed -11.0…

I have no idea how this software has been developed, if they have done any sort of testing, but the addition of features and apparent bugs indicate a rushed project without input from people that will watch the event on “real” TV sets.

But since Björn Ferry won gold, I am not upset anymore; rather this software can be seen as a nice opportunity to practice your ability to see anomalies in real time software.

A disclaimer… I am no developer, but I have developed a tool. As I develop I have the mindset of a developer, not the tester. I have done lots of mistakes, intentionally not implemented good/needed things and considered what parts I can get away with in the first release. This tool might not seem big and useful, but I have used it and it has created many interesting results in the past. As I developed this I tried a new method of implementation… all ideas I had on what functions the tool should have, what was supposed to work, what was not supposed to work etc I wrote down in a testideas-document. I then had one column that identified if it worked or not in a specific release. All good feedback I added to that list.

This is the first tool we create at the test eye that is open for the public. At thetesteye we have choosen to publish our material under the license Attribution No Derivatives. My personal aim with this tool was to increase my knowledge of coding. I have used Python and Tkinter as a graphical interface. In the Publications section you will find the link to the tool and the currently released version.

General discussion

The general idea is to use the frequency of words as a way to find errors. The more text you analyze, the higher statistical significance; thus resulting in an easier chance of spotting the erroneous words. This kind of script is very often found as a code example. When I first created a script for this I did not know that. I ran it on a quite large text corpus and found that the company name had been spelled incorrectly 7 times in the copyright text. I also found lots and lots of spelling mistakes as well as some strange API functions that were incorrect.

Use cases

• Run on documentation to find unfrequent words (that usually contains spelling errors)
• Run on code to find variables that are similar but not the same and used incorrectly
• Run on code to find unused variables, thus variables only used once
• Run on code + API documentation to find things that should not be there or code that are not covered anywhere
• Localization specific: When doing translations you might be allowed to have a certain amount of errors, this is one way of finding a few extra faults that you can remove

How I use it

I run the tool on a tree structure. I open the result file in Excel or OpenOffice Calc. I then sort on frequency… start deleting uninteresting records. You can open it in MS Word or something similar to filter out things that are in fact spelled correctly. After a few cleaning ups you might have a list that is worth investigating.

Bugs and Enhancements

The testideas.xls contain the current tests and some of the enhancements that I’ve gotten so far. If you got any suggestions, feel free to mail me at martin.jansson@thetesteye.com.

In my local food store they have this system where you scan the price tags on the food you buy and most often smoothly able to pay and exit without having to stay in any long queues.

A time back they must have changed software in these scanners because their behavior changed and bugginess increased. The funniest bug or feature, as they themselves would most certainly call it, is when you are finished. You then scan a finish code which then sends a signal to the scanner, then you are able to pay. When you perform this last scan you will now hear a loud beep from the device, previously this beep was used when there was an error of some kind. So, everyone (at least everyone who I have seen do this) perform the last scan and upon hearing the beep they exit the queue and go to the cashier. The cashier then tells it is supposed to sound like that and that it is perfectly normal. This happens every time for everyone, at least once for us technocrats… but prolly each time for those who are a bit scared of technology still. One of the main ideas with using this device is to minimize the effort for the cashier by letting customers check-out on their own. This sound stops that feature.

Another funny bug that have appeared quite recently is that it takes a bit longer to scan wares, I mean from less than a second to close to ten seconds per ware. Pulling up a box of tomato sauce where you must scan each of the 12 cans will now take about two minutes… you just stand there continuously pressing scan… waiting and building up on that hysterical laughter. The idea of Shop Express has lost a bit of its flavour, still when it works as expected it is indeed a lot better than using the normal queue.

I just want to promote a really good blog post written by Michael Bolton where he describes the difference between Testing and Checking:
http://www.developsense.com/2009/08/testing-vs-checking.html

I wish that many managers, testers and developers read this post…

Cheers,
Henrik

The front door

The bug that a co-worker found was that you were able to pull open the door a quarter of an inch, just enough to be able to put in a little stick and then wave it infront of the radar. It actually goes quicker to use the stick instead of the card.

I confronted the installer of the security radar and told him about the bug. He answered me “No, that is not possible.”. I told him again, it is so and that one of our consultants use it since he has no pass card. He answered me again “No, that is not possible.”. I did not have time to argue so I left.

I am now going to notify the owner of the house so that they understand the problem. Apparently there are more people using this trick when they have forgotten their card, so the stick is just below the pass card holder for easy access. A few years ago there was something called “Not my job award!”, I guess this would fit into this category.

Random testing or fuzzy testing is nothing new, but for those of you who are new to it I just wanted to share a little tool I found. If you want to know a bit more about fuzzy testing go read at http://en.wikipedia.org/wiki/Fuzz_testing or whatever place you like to find quick info at.

Barton Miller has written a bit about this and made some binaries to use at:

ftp://ftp.cs.wisc.edu/paradyn/fuzz/

Read ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-nt.pdf for interesting MS Windows results.

The binary subjects an application to a stream of random input either by keyboard and mouse events or by using SendMessage and PostMessage. I used the binary on WinXP on UltraEdit and MS Word. I even tested the feature to simulate mouse click and mouse movement with a few interesting results.

What does this tell me? What would I expect from this? Is this a tool that I want to have in my tool box when testing software?

Using this tool will help you check robustness of an application. It will do automated random testing. If an application crashes often it might be that they have not properly handled Win32 messages. I will try this out in my toolbox. Even if you decide not to use it, read Bartons article.

At my former company we had a vision of removing/fixing all unhandled exceptions that we found (and that we thought a customer might encounter). After some discussions we decided that the unhandled exceptions should be treated in the same way as we had treated severe errors on previous platforms. I.e. Crashes in C++; NullPointerExceptions in Java; “Runtime Errors” in VB; and “Script Errors” in JavaScript.
This was a philosophy that both testers and programmers agreed upon (more or less) and it fitted our context.

Now, for the last two years I have been working as a test consultant; and during these years I have been on a couple of customer sites where their applications have been developed in C# .Net.

When unhandled exceptions occur, many team-members and users that I have been observing just click “Continue” and continue with their work as if nothing happened. And when I afterwards have asked them why they did so, it appears that they never reflected upon the error and if it might have been severe.
Many say that they don’t see the unhandled exception as a crash, or “Soft Crash”, instead they have seen it as a warning or an error that “really shouldn’t be in there”; some people see it as an error that in fact is handled, hence the error message and the pop-up dialog box.
Sometimes I point out that after an unhandled exception you cannot be sure about the state that the application is in, and that any behavior afterwards might be affected by the first “crash”. But I rarely hear anyone else saying this.

I am not saying that hard crashes are to be preferred in any way, but then you at least understood that something was severe.

I think that soft crashes give users/developers/testers/stakeholders/etc a false sense of safety.
And I blame the “good handling of unhandled exceptions” that the .Net framework provides. In fact, I blame the “Continue” button of those dialogs.

What are your experiences and thoughts regarding this?

(This was originally posted as a new thread on the Context-Driven Software Testing list:
http://groups.yahoo.com/group/software-testing/message/5794 )

The other day I was using Google Chrome to browse through internet.

One thing I did during that session was to try to download an evaluation of HP Quality Center. I was struggling to register as a user and try to download, but it was not possible.

But how does this relate to Google Chrome? I am not sure if this might have to do with one of their EULA-paragraphs about google owning all data sent to their server:
The day after I had the attempts to download the demo, a woman from IBM called my cell phone. And what was she offering? You might have guessed it by now?

She wondered if she could tell me more about the IBM Rational Testing suite.

Was this just a coincidence? I don’t know…