Here are some small, free, nifty tools I use now and then:

FreeMind – to model and communicate
WinMerge – to diff or merge files or folders
Process Hacker – to monitor resource usage
Process Monitor – to monitor registry and disk activities
InCtrl5 – for installation testing (what happended to Install Analyzer??)
Fiddler/Wireshark – to see network traffic details
Firebug – to see and edit details in Firefox
Xenu’s Link Sleuth – to find broken links
Zed Attack Proxy – lightweight security testing
NetLimiter – to simulate low bandwidth
Om/Perlclip – put testdata in Clipboard
Rapid Reporter – to document your testing
Color Oracle – to simulate color blindness
Console – a better command prompt

Additions are welcome!

I recently acquired the best product in the world, for me.
It is a Victorinox serrated multi-purpose chef’s knife that I use to slice bread.
In 2000, chefs around the world voted it as the best knife in the world.
I want to go one step further, and name it the best product in the world, all categories.
Some people say that if you go to a restaurant, and they don’t have this knife anywhere, you might as well leave at once.

When I use the knife it has an incomparable feeling when it cuts through the hardest crust of a sourdough bread. There is an ease in its operation that brings well-being.
It might have drawbacks I haven’t noticed yet, but with this 100% Charisma I don’t really care, I could slice bread for days.

So what kind of charisma does your product have?
Can you test for this and make suggestions for improvements?

This began as an answer to Rikard’s post http://thetesteye.com/blog/2011/06/a-word-of-caution/ where the discussion on “traditional testing” came up.

I often hear comparisons with our “industry” and the Automotive industry.
In that context, you could say that “traditional testing” corresponds to the methods and practices that are applied in line production of large car companies. And the unorthodox testing can be compared with those specialized and often smaller custom car builder companies out there.

The major issue with this kind of comparison is that large car companies makes thousands of the exact same type of car. Instead this means that the “traditional testing” approach is an attempt to apply line production methods when building custom cars. Applying “traditional testing” as if every project and product were the same is both wrong and dangerous.

And this comparison is not that far-fetched… It seems to me like “traditional testing” is promoted as practices that suits many (if not all?) projects and should be followed in order to enable success. Well, good luck!

Further, many practices that we use today comes from the automotive industry (at least in their latest form).
If we fail to see why they implemented them in the automotive industry and just take them as good practices for being effective in line production, we are doing the opposite of what the automotive industry did. They did some investigation on how they could improve their work. Some of that included seeing the human beings as intelligent and social creatures; utilize the diversity of a group of people; etc. And their productivity and efficiency could be measured by measuring number of cars and their quality. So by treating humans as humans they became more efficient (number of non-defective cars) and improved their work methods (analyzing quality of work).
When Lean development is implemented in psychiatric nursing or software development today, the focus is very often on the quantity measuring part which then misses the whole point. Measuring patients or software as uniform units is very wrong and dangerous.
It’s not that Lean or Kanban is to blame, it’s the implementation; and perhaps mostly the implementors.
The role models for Lean implementations in many healthcare institutions in Sweden have been some successful nursing teams that have increased their efficiency and quality of their work by using Lean development as a method. What those successful teams really did were to take command of their own work and found a method that supported their initiative and commitment. (Anyone had similar experience? Me!)
The problem is that when this is implemented by management at the whole hospital or nationwide, the focus is shifted from “Quality of work” to “Quantity of work” because that is the obvious driver for management and really the only incentive for they to implement it. They only need to say that this is a “best practice” and then it’s OK…
I do need to emphasize that you don’t automatically get high quality work by implementing “best practises”!

This is happening all the time; and the last flavor of the month is Agile.
If you read the Agile Manifesto and then think that you must play planning poker or have standup meetings you are obviously not understanding the Agile Manifesto.
I’m with Matt Heusser on his interpretation here: http://www.softwaretestpro.com/Item/5036/My-%27take%27-on-Agile—-or-the-Manifesto-elaborated/Testing-Software-Test-and-QA-Teams-Strategy-Agile-Development

The tool you’ve been waiting for!

Function: When you push the pedal a random input will be sent to the machine, and thereby your application.
By default, a sample of error-prone inputs are available (e.g. ASCII 30, double-click, Unicode, beep)
The nifty thing is to be able to do this rapidly, On-Demand, in unexpected situations, or when you are bored.

System Requirements: any machine with USB (Wi-Fi in 2.0)
Modes: ASCII, all characters, everything
No of inputs sent: 1 to 4,294,967,296
Logging: Temp folder on applicable systems
Limitations: no API, no warranty
Price: TBD

If you plug it in to someone else’s computer, you can take your practical jokes to a new level…

Here is neat story I told to Michael Bolton, Martin Jansson and Markus Gärtner when we were exploring the Metro in Copenhagen, during the coldest days the city had experienced since they started measure the temperature. I promised to blog about it…

Let me begin with some background information.

A couple of years ago I worked as a tester in a project where we developed an application that did an automated processing of an electronic version of a paper form.

In many ways this was an interesting system – no GUI, no user input; no visible output (if everything went OK). And the paper form was something most people in Sweden would use at least once; so it was very important that the system did things right. In fact, I used it myself just days after my assignment was over.

In short, this is what the system did:
My organization sent out a paper form regarding a matter to people with some hard-coded text in it (e.g., names and personal identity number, etc), and people should fill out the rest of the form.
They would then send in the paper form to a third-party company that used OCR to convert the paper form into an electronic form (xml-file). The file was then sent to my organization and went into our system, where the processing of the form content took place.
First there were many checks done in order to check the accuracy and format of the manually entered text. If that passed, it went on and did checks against all the laws that were supposed to be met in order for the matter to be processed correctly; if that passed it went on and a formal decision could be made, which included notifying several organizations and the people who sent in the paper form in the first place.
If any format check or law was violated, it went to a manual handling of the matter.

The test data we used was non-real people, but they still had to be unique in the national database system (so if our database had included all existing persons, our non-existing persons would have been unique persons in that sense). So we only got 2 test persons a week, because that was what could be reserved for testing purposes. This meant that we needed to be very careful with these persons and design them with utmost precision in order for them to not violate any rule in an unintended way and thereby be caught by the system. And vice versa, those who should be caught by a certain rule needed to just be caught by that rule and nothing else, even if that meant that an earlier check might have been introduced later in the project.
So you might understand that we put a lot of effort in the test data and making sure that it would be valid.

Anyway, this story is about what happened at the end of the project.
After we had developed all tests and they ran through the system successfully – meaning that many should be caught by the system, and many  should run through all the way without being caught – it was time for us to take the tests one step out of our system. This meant printing out all paper forms; and manually write the text that should be included; and then send them to the third-party company that would scan the paper forms and convert them into electronic forms which they then sent into our system. The intention was that the result would be the same as when we ran the electronic forms. Partly why we did this was because the third-party company had trimmed their OCR-machine in order for it to understand this new paper form; so we wanted to know how good work they had done with this. All their reports said that the results were OK; and we had sent them a couple of forms and we were also satisfied with the result. But we wondered if the OCR-system could handle some tricky data (which obviously some of our tests were designed to be).
So we began filling out the forms according to our test cases. E.g. one format check could be to make sure that only one word was written in a field, so the test included a word with a space inside – which we then wrote as obvious as possible in order for the OCR-machine to interpret this as two words and thereby it should be caught.
There were plenty of these format checks that we carefully tried to violate. Another example was to write with a really hard handwriting style so that it would be really hard for the machine to interpret and thereby flag the field as “unreadable”. Etc.
At the end of the week, we sent all these papers with a box delivery firm to the third-party company and waited for the forms to drop into our system on Monday morning. We were all excited because this really felt like a proper production test with test data as close to the real stuff as possible.

On Monday morning, the first forms were dropping in and processed by our system. We monitored the process by following them through the database and all the states that they ended up in. To our surprise, most of them passed all the format checks and went further into the system… We started out investigating the xml-files and the scanned tif-images. The tif-image showed our paper form and they were correct; but the xml-files didn’t get the flags that we would have expected. Hmmm, strange…
Our reaction was: “What an amazing OCR-machine! How can it interpret so good!?”

We reported to the third-party company that we weren’t satisfied with the tests; and we told them that we would send them a new batch as soon as possible. We didn’t say why we were unsatisfied, because we thought that we had screwed up and not exercised the OCR-machine to the limits.

So we began the tedious work of filling out the forms again; but now even more evil then before.
Now, the OCR-machine shouldn’t stand a chance against our cruel intentions.
As we did the previous week, all papers were shipped to them at the end of the week and on Monday we were back and rubbed our hands waiting for the forms to enter the system.

But the same thing happened the second time!

Now I took a tif-image and the corresponding xml-file and went to one of the business analysts. “How the hell can the machine interpret this garbage text into something as useful as what it says in the xml-file?”. We looked at it and shook our heads. We couldn’t believe that this was happening.
I went back to the system and analyzed the data some more.

Suddenly I discovered a typo in a name and instantly got suspicious. I recognized the name since I had created it when creating the test person. Something was very wrong here…
Then it hit me! The name had been pre-printed on the paper form and shouldn’t have caused any trouble for the OCR-machine given the results for the handwritten stuff. I thought to myself “It’s a human behind this!”.

I went to the business analyst again and told him about this. He said “Damn, that’s it!”
He then called the third-party company and asked to speak with the person responsible for the OCR-machine. He then asked “Do you know if someone has interpreted some of our paper forms manually?”
The answer dropped as a bomb:
“Well, yes. All of them. We’ve had some trouble with the machine so we had to do it manually. And I want to say that it was really hard for us; you had written in such bad handwriting that it took us so much time to process them that we thought that this was torture. And just as we thought that it couldn’t get worse, the second batch came in that was way worse than the first one. We had to sit in pairs and process them carefully and with utmost respect. But we did a hell of a job, don’t you think?”

I came to think about this when I read Jerry Weinberg’s “The Secrets of Consulting” and saw The Helpful Model:

No matter how it looks, everyone is trying to be helpful.

Monday

Henrik started his journey by car from Karlstad and went down to Gothenburg to join with Martin. Both of us were going to take the train down to Copenhagen. Not surprisingly there were delays and the train was cancelled… Instead we headed back to Martin’s house and loaded Henrik’s car and took off. After several hours drive we arrived to Copenhagen, where we went directly to Bella Center and the TestLab at EuroSTAR.

This was the first time we met James Lyndsay and Bart Knaack in person; we had only talked with them over Skype/phone. So we all started with saying hello but there were really no time to sit down and have a cup of coffee – we needed to get to work straight away.

One of the burning issues to resolve the week before the conference was to bring client machines. Henrik was able to bring four laptops (thank you Compare Testlab and Know IT for sponsoring with this!) and Martin one. We worked in the TestLab for a few hours setting up and testing the two networks (James had one and Bart had another) with the servers connected to them.

Each network consisted of a bug tracker (Mantis), a server with OpenEMR, a server with a WebShop based on OsCommerce as well as a link to download a version of FreeMind.

When the conference center was getting near closing hours we rounded up and identified tasks needed to be done for the next coming days. We were assigned to setup and fill both the OpenEMR and the WebShop with new test data. We determined that we first should create a data model in FreeMind; both for reviewing before inserting the data and also so that we could print out the model and use it as a means to discuss with participants in the TestLab. With the model at hand, we thought the participants would also be able to identify test ideas of their own that they could try out.

We packed up everything and took Henriks car together with James and Bart. Half an hour later we were down in the hotel bar for dinner. We first met up with Steve Öberg, Rikard Edgren, Björn Karlsson, and Stefan Thoresson; and later on Markus Gärtner, Neil Thompson & Michael Bolton and some more joined our table and we were discussing testing long into the night.

Tuesday

At 8.45 we met up with James and Bart at the hotel and took a cab to the conference center where we setup the TestLab again. We started on the data models while James and Bart worked on setting up the details of the two networks. Meanwhile the sponsors were installing their applications on the client machines. Late into the afternoon we were starting to get ready with the new data for both systems. During the afternoon several people were visiting us and some of them started testing the applications. In fact, the first bug entered in the bug system was the one that won the prize “Best bug” – Markus Gärtner found it pretty fast…
Around 18:00 we were able to move into the room where the TestLab should be. We moved in all servers, routers and clients and setup the power for it all. Funny thing is that we didn’t find more than one electricity socket, so all machines were hooked up on this only one and we decided to load test the electricity over night.
During the evening we attended the Rebel Alliance (or Danish Alliance) for 11 interesting lightning talks. The discussions went on after that long into the night. But to our defense we must say that we got in bed in time.

Wednesday

Despite that we got in bed fairly early, we both overslept and missed the grand opening of the TestLab… Such a good start!
But no time to cry over that; we dug in and assisted in welcoming and introducing people that entered the TestLab and told them about it and how to use it; as well as guiding them to start testing. Bugs were reported and the sound of the chicken echoed all over. Nice!

During the day a lot of people were in and really sat down testing. There were also a lot of interesting collaboration going on between people that hadn’t met before. Some of the people that spent more time than others were Shmuel Gershon, Markus Gärtner, Isabel Evans, Michael Bolton, Ajay Balamurugadas, Zeger Van Hese, and Rob Sabourin (we might have forgot some of the names, but you know who you are).

During the day there were a lot of non-arranged  testing going on. But we had two really appreciated sessions during the day.
Firstly, there was a pair-testing session led by James Lyndsay where close to thirty people attended. There was a good mixture of experienced and less experienced testers. Even though the session was only about 30 minutes divided in three 10 minutes mini-sessions, it seemed to give the participants some very nice hands-on experience to bring home. And a lot of good collaboration was happening!

Secondly, during lunch hour it was an improvised weekend-testing session, Ajay Balamurugadas had invited as many as possible during his talk on Weekend Testing right before the session. Ajay and Markus Gärtner led the session with the rest of the participants that had showed up. Since there were so many that showed up we teamed up in pairs or smaller groups. The collaboration in the TestLab was yet again really great! After close to 30 minutes session we had found a bunch of interesting bugs while at the same time many have had the opportunity to team up with these fabulous  testers.

Despite that we had to cancel some planned sessions, we all thought that the first day was a real success . And we measured that by realizing that we didn’t have had time to take breaks or eat a proper lunch because the TestLab wasn’t empty for even a minute; there were at least 5 people there at minimum (except during the key note by Stuart Reid).

On the evening there was the Gala Awards dinner at Copenhagen City Hall and we ended up at a pub called Bryggeriet with the Rebel Alliance discussing testing in all forms. We went home pretty early and met up with some people in the hotel and continue discussing testing.

Thursday

The day kicked off at 08:00 in the TestLab and we used the first 30 minutes preparing for yet another interesting day!

At the AllSTAR testers, a few renown testers were able to join in. We split the TestLab into three groups where each focused on a specific area which they knew well. Each group attacked the systems in various ways, teaching the participants new ideas on how to test the systems. Ari Takanen was showing how he did some security testing (and/or fuzz testing); Rob Lambert showed how he did accessibility testing; and James Lyndsay had a exploratory testing session discussing a lot of test ideas with the participants.

At the round the table of managing ET, James Lyndsay moderated the group consisting of Rob Sabourin, Michael Bolton, Carsten Feilberg, Shmuel Gershon, Zeger Van Hese, Henrik Andersson, Henrik Emilsson, Martin Jansson, Markus Gärtner, John Stevenson, Neil Thompson and some more (we are sorry if we missed to mention someone, but we try our best to remember all people that were there out of those 200 TestLab attendees). We talked about several aspects around managing ET; and the discussion began with how we had managed our own time in the TestLab. James Lyndsay moderated the group in an excellent way. One of the subjects was how often you change/update your charters. Another interesting topic was on how much information you should provide different testers with.

During the lunch session, Markus Gärtner launched a Testing Dojo that was very appreciated. And Michael Bolton was doing a transcription of the session. The testing dojo meant that one tester tested the application with a projector that showed the computer screen. The tester then described through the testing and explaining what happened. All other around the tester could ask questions. The tester should be switched every 5 minutes letting several people show off their skills.
While this was the intention, the testing dojo slightly turned out to something different. But still as much appreciated by those involved.

Thursday evening we, Markus Gärtner, and Michael Bolton headed in to Copenhagen for a dinner with Ajay Balamurugadas, Teemu Vesala, Kalle Huttunen and Mika Hänninen. It took us two hours to get to the restaurant because we were forced to explore the Copenhagen Metro and train system…

Friday

We headed home to Sweden in the morning and arrived in Gothenburg at 15:00 in the afternoon. Then Henrik continued to Karlstad and arrived at 18:30. Seeing the problems other people had with getting out of Denmark by plane or train, we were lucky that car traffic wasn’t too affected by the blizzard…

Some numbers

• 180 people got in to the TestLab and got introduced to what we do (most of them sat down and tested)
• 100 hours of testing
• 100 reported bugs (plus several bugs not reported in the bug system)

Reflections

Considering how the TestLab was run, it was like a mix of having an open house and being a test lead handling several projects at the same time. We served different stakeholders such as the participants in the lab, the speakers, the conference personal and ourselves as apprentices as well as the masters (James and Bart). The plan is ever changing. Each stakeholder value different things. Just like in the real world outside the TestLab.

There were also 5 sponsor presentations from the tool vendors that had their tools on the client machines. We really appreciated those sponsors that had put in some time to really show how their tool could be used on the same applications as we tested in the TestLab. Very useful and meant that there were more value for those interested in the tools to try things out for themselves on the lab machines.

During the TestLab we got a lot of ideas on what we want to do next year. A lot of ideas that we had on beforehand were though only good in theory; seeing the real obstacles and problems in practice was a good lesson.

And finally, we really appreciate the great effort that James and Bart have done in creating the EuroSTAR TestLab and made it to such a remarkable success. We will have the benefit of building upon a very solid and well thought-out platform & concept!

We had a lot of fun and hope that all others had a great time!
See you all in Manchester 2011!

Cheers,
Henrik and Martin

That applications have focus on the right place is essential to a good user experience.
You have to trust that pressing Del on keyboard will have the intended effect.
Problems with this is very common, at least on Windows, and especially in applications with dialogs and panels and stuff of different types.
Addressing symptoms agressively easily ends up with flickering dialogs, and sometimes worse user experience.
Less is less, but it has to be the right code…

Focus can also cause miscellaneous functional problems;
this is my latest favorite example:

Client: Windows Vista or Windows 7, .NET 3.5, Office 2007

1. Launch Word 2007
2. Type any text
3. Add a Footer, and add any text
4. Put focus on “main text”
5. Press Ctrl+F and search for something you wrote in Footer
6. When the footnote is found, click on the Footer title bar (this is the crucial step!)
7. Click “Find Next” again
8. Click OK in dialog “Word has finished searching the document”

Result: Word crashes.

Expected: That I am able to continue writing my document.

I saw an interesting interview with Rob Sabourin today http://www.youtube.com/watch?v=HZRXdaN7gkY  (Thanks for the tip, Jon Bach!)
One thing he says in this video is: “… There are a lot of template junkies out there. [Testers are]  filling out templates and not actually testing. That frustrates testers…”

Hey, isn’t this the same thing that happens in strictly scripted testing environments?
When testers are following detailed test scripts they are actually ending up filling out templates. With the same frustration as when filling out templates where you also don’t have to bother use your brain.
The result is the same for scripted testing as with a filled out template. Little mind work has been involved and so the information is very narrow and/or shallow…

It is sometimes said that templates are good because it is prevent someone from doing mistakes and/or making sure that the format is the same all the time. But isn’t a template to a document the same as what a script is to a test? A prescribed recipe on how to complete the task. I.e. it is about management control.

Isn’t it so that you easily can be hit by inattentional blindness as you fill out the template and only worrying about not screwing up the format? As with scripted testing, the focus lies in the format rather in the content. Both in scripted testing and in filling out templates you worry about making sure all things are filled out as expected, not the actual testing or the actual writing of informative text.

And, as with scripted tests, if we could fill out the templates automatically, we wouldn’t need to do them manually.

Last week, me and Martin won the competition “The EuroSTAR Test Lab Apprentices”!
Read more at: http://www.eurostarconferences.com/delegates/the-test-lab-apprentice.aspx

See you in the Test Lab in Copenhagen!

Cheers,
Henrik & Martin

As a user of Facebook I feel really helpless when nothing works as it should (as was the case with the latest GUI-update). Posts were stochastically shown in the feed and a lot of errors occurred in various situations. A lot (all?) of my friends on Facebook experienced the same problems.

When there are lots of bugs on flight booking sites I get so frustrated and angry because I cannot complete my task. E.g., the booking system for Ryanair (at least two years ago when I last used their booking system and never will use again).

But what can I do as a frustrated enduser?

The problem nowadays is that you as a user and consumer don’t know what to do with your complaints. The companies shows no respect as long as the users are hooked on the application and needs the solution to problem that the application provides. But they do not care to fix the bugs that might ranging from annoying to critical.
So the loser is you! And it becomes a rage against the machine because of the absence of real people to talk to. Or some people might think that they did something wrong when it really was lousy software that caused the problems.

But in fact all other users of these applications experience similar problems. So you as an enduser is not alone! This is something that hits many people on their own but who lacks a community i.e., if we were employees in a company using the software we would have protested.

What should we do!?

Should we create a site similar to http://www.badsoftware.com/ (Kaner & Pels) or http://www.badsoftware.com/alienwaresucks/ (Kaner) in order to channel some of the frustration and gather it into some powerful criticism?

Or are there other channels that we could utilize?

Do you have any ideas?