I have for a long time felt that something is wrong with the established software testing theories; on test design tutorials I only recognize a small part of the test design I live in.
So it felt like a revelation when I read Gerd Gigerenzer’s Adaptive Thinking where he describes his tools-to-theories heuristic, which says that the theories we build are based, and accepted, depending on the tools we are using.
The implication is that many fundamental assumptions aren’t a good match for the challenges of testing; they are just a model of the way our tools look.
This doesn’t automatically mean that the theories are bad and useless, but my intuition says there is a great risk.

Software testing is suffering a binary disease.
We make software for computers, and use computers for planning, execution and reporting.
Our theories reflect this much more, way much more than the fact that each piece of software is unique, made for humans, by humans.

Take a look at these examples:
* Pass/Fail hysteria; ranging from the need of expected results, to the necessity of oracles.
* Coverage obsession; percentage-wise covered/not-covered reports without elaborations on what is important.
* Metrics tumor; quantitative numbers in a world of qualitative feelings.
* Sick test design techniques, all made to fit computers; algorithms and classifications disregarding what is important, common, risky, error-prone.

When someone challenges authorities, you should ask: “say you’re right, what can you do with this knowledge?”

I have no final solutions, but we should take advantage of what humans are good at: understanding what is important; judgment; dealing with the unknown; separating right from wrong.

We can attack the same problems in alternative ways:
* Testers can communicate noteworthy interpretations instead of Pass/Fail.
* If we can do good testing and deem that no more testing has to be done, there is no need for coverage numbers.
* If the context around a metric is so important, we can remove the metric, and keep the vital information.
* We can do our test design without flourishes; focusing on having a good chance of finding what is important.

Do you think it is a coincidence that test cases contains a set of instructions?
These theories cripple testers; and treat us like cripples.

Now you know why people are saying testing hasn’t developed the last years: we are in a dead end.

And the worst is that if Gigerenzer is right; my statements have no chance of being accepted until we have a large battery of tools based on how people think and act…

We, testers, are we all alike? Looking back over the years I’ve been involved in testing, I would say I have met a whole bunch of different testers. Some shared my passion and fascination for testing, others were not testers they just happened to have the job title. However, I have always appreciated our craft for having such diversity of people. We have testers that have strong domain knowledge, programming skills and business knowledge. All of those are easy to connect the value to have in a test team, they find important information about the technical implementation of the product.

Then we have the other group, the ones who have capabilities in philosophy, cognition, history, economics, art, music and myself a drop out from statistics at university. We are not here because we love code or the coolest technologies, we are here because we love the way people and products interact and misunderstand each other. We are fascinated by the dance between a human and a machine. We find patterns in this dance, we find unintentional behaviors, and the unexpected surprises us. We look beyond constraints of the application, we go where no developer intended to go. By doing this we notice other kinds of things that might be a threat or a problem for the success of the product. Why are we doing this you may ask? It is because the technical solution does not impress us, we do not take particular interest or care in what specific way the code is written. What drives us is the question: will this solve a problem for a user and in which ways can it fail to do so? What we bring to the table is the social and behavioral science aspects of the product combined with the human element. At technology facing companies I sometimes see the above overlooked or under-valued.

But what worries me more is that we, testers, are walking away from my precious diversity. Strong forces favor uniformity among testers and the most obvious one is ISTQB. After going through their program of various certifications everyone knows the same, speaks the same language and uses the same templates. There is no problem replacing one ISTQB Advanced Level certified test manager with another one with same level of certification. The same goes with TMap, as long as you can read the book you know the answer to all testing problems, it’s easy!

But this is nothing new, I and many with me have been fighting both ISTQB and TMap for years. But here comes the kicker, have you heard of this cool new thing called “agile”? It’s a really sweet thing, close to a drug in our craft. It came and it has conquered. Nowadays there are almost no companies who dare say that they are not “agile”, instead they make the most ridiculous variations of what they call “agile”. Don’t get me wrong here, this new movement is quite appealing to me and I appreciate much of what it stands for.

Now getting back on track there is one thing that worries me about this agile thing. That is the view on us testers. Now to be an “agile tester” you basically need to be a developer with preferably some basic knowledge of testing, but it is much cooler if you know Selenium, TDD, ATDD and other strange abbreviations. If a tester according to the agile folks is basically a developer, why do you call them a tester, or wait you don’t! Everyone is a team member, I’m sorry.
Now we are walking from “must have a certification” to “must have programing skills”. Guys, don’t you see what is happening. We are just replacing one uniformity with another, beware of this!

We are running a great risk that all our testers are finding the same information and worse, are blind on the same spots. In testing we do not know what information we will get and we do not upfront know what questions to ask the product (don’t be mislead to believe anything else). We have to approach the testing from various different angles and have several different ears and eyes observing it. If one thing, we should love diversity among testers and we should encourage it.

This post came to my mind when I once again listened to Malcolm Gladwells brilliant TED Talk.

And no we do not all want black coffee!

Here is neat story I told to Michael Bolton, Martin Jansson and Markus Gärtner when we were exploring the Metro in Copenhagen, during the coldest days the city had experienced since they started measure the temperature. I promised to blog about it…

Let me begin with some background information.

A couple of years ago I worked as a tester in a project where we developed an application that did an automated processing of an electronic version of a paper form.

In many ways this was an interesting system – no GUI, no user input; no visible output (if everything went OK). And the paper form was something most people in Sweden would use at least once; so it was very important that the system did things right. In fact, I used it myself just days after my assignment was over.

In short, this is what the system did:
My organization sent out a paper form regarding a matter to people with some hard-coded text in it (e.g., names and personal identity number, etc), and people should fill out the rest of the form.
They would then send in the paper form to a third-party company that used OCR to convert the paper form into an electronic form (xml-file). The file was then sent to my organization and went into our system, where the processing of the form content took place.
First there were many checks done in order to check the accuracy and format of the manually entered text. If that passed, it went on and did checks against all the laws that were supposed to be met in order for the matter to be processed correctly; if that passed it went on and a formal decision could be made, which included notifying several organizations and the people who sent in the paper form in the first place.
If any format check or law was violated, it went to a manual handling of the matter.

The test data we used was non-real people, but they still had to be unique in the national database system (so if our database had included all existing persons, our non-existing persons would have been unique persons in that sense). So we only got 2 test persons a week, because that was what could be reserved for testing purposes. This meant that we needed to be very careful with these persons and design them with utmost precision in order for them to not violate any rule in an unintended way and thereby be caught by the system. And vice versa, those who should be caught by a certain rule needed to just be caught by that rule and nothing else, even if that meant that an earlier check might have been introduced later in the project.
So you might understand that we put a lot of effort in the test data and making sure that it would be valid.

Anyway, this story is about what happened at the end of the project.
After we had developed all tests and they ran through the system successfully – meaning that many should be caught by the system, and many  should run through all the way without being caught – it was time for us to take the tests one step out of our system. This meant printing out all paper forms; and manually write the text that should be included; and then send them to the third-party company that would scan the paper forms and convert them into electronic forms which they then sent into our system. The intention was that the result would be the same as when we ran the electronic forms. Partly why we did this was because the third-party company had trimmed their OCR-machine in order for it to understand this new paper form; so we wanted to know how good work they had done with this. All their reports said that the results were OK; and we had sent them a couple of forms and we were also satisfied with the result. But we wondered if the OCR-system could handle some tricky data (which obviously some of our tests were designed to be).
So we began filling out the forms according to our test cases. E.g. one format check could be to make sure that only one word was written in a field, so the test included a word with a space inside – which we then wrote as obvious as possible in order for the OCR-machine to interpret this as two words and thereby it should be caught.
There were plenty of these format checks that we carefully tried to violate. Another example was to write with a really hard handwriting style so that it would be really hard for the machine to interpret and thereby flag the field as “unreadable”. Etc.
At the end of the week, we sent all these papers with a box delivery firm to the third-party company and waited for the forms to drop into our system on Monday morning. We were all excited because this really felt like a proper production test with test data as close to the real stuff as possible.

On Monday morning, the first forms were dropping in and processed by our system. We monitored the process by following them through the database and all the states that they ended up in. To our surprise, most of them passed all the format checks and went further into the system… We started out investigating the xml-files and the scanned tif-images. The tif-image showed our paper form and they were correct; but the xml-files didn’t get the flags that we would have expected. Hmmm, strange…
Our reaction was: “What an amazing OCR-machine! How can it interpret so good!?”

We reported to the third-party company that we weren’t satisfied with the tests; and we told them that we would send them a new batch as soon as possible. We didn’t say why we were unsatisfied, because we thought that we had screwed up and not exercised the OCR-machine to the limits.

So we began the tedious work of filling out the forms again; but now even more evil then before.
Now, the OCR-machine shouldn’t stand a chance against our cruel intentions.
As we did the previous week, all papers were shipped to them at the end of the week and on Monday we were back and rubbed our hands waiting for the forms to enter the system.

But the same thing happened the second time!

Now I took a tif-image and the corresponding xml-file and went to one of the business analysts. “How the hell can the machine interpret this garbage text into something as useful as what it says in the xml-file?”. We looked at it and shook our heads. We couldn’t believe that this was happening.
I went back to the system and analyzed the data some more.

Suddenly I discovered a typo in a name and instantly got suspicious. I recognized the name since I had created it when creating the test person. Something was very wrong here…
Then it hit me! The name had been pre-printed on the paper form and shouldn’t have caused any trouble for the OCR-machine given the results for the handwritten stuff. I thought to myself “It’s a human behind this!”.

I went to the business analyst again and told him about this. He said “Damn, that’s it!”
He then called the third-party company and asked to speak with the person responsible for the OCR-machine. He then asked “Do you know if someone has interpreted some of our paper forms manually?”
The answer dropped as a bomb:
“Well, yes. All of them. We’ve had some trouble with the machine so we had to do it manually. And I want to say that it was really hard for us; you had written in such bad handwriting that it took us so much time to process them that we thought that this was torture. And just as we thought that it couldn’t get worse, the second batch came in that was way worse than the first one. We had to sit in pairs and process them carefully and with utmost respect. But we did a hell of a job, don’t you think?”

I came to think about this when I read Jerry Weinberg’s “The Secrets of Consulting” and saw The Helpful Model:

No matter how it looks, everyone is trying to be helpful.

Monday

Henrik started his journey by car from Karlstad and went down to Gothenburg to join with Martin. Both of us were going to take the train down to Copenhagen. Not surprisingly there were delays and the train was cancelled… Instead we headed back to Martin’s house and loaded Henrik’s car and took off. After several hours drive we arrived to Copenhagen, where we went directly to Bella Center and the TestLab at EuroSTAR.

This was the first time we met James Lyndsay and Bart Knaack in person; we had only talked with them over Skype/phone. So we all started with saying hello but there were really no time to sit down and have a cup of coffee – we needed to get to work straight away.

One of the burning issues to resolve the week before the conference was to bring client machines. Henrik was able to bring four laptops (thank you Compare Testlab and Know IT for sponsoring with this!) and Martin one. We worked in the TestLab for a few hours setting up and testing the two networks (James had one and Bart had another) with the servers connected to them.

Each network consisted of a bug tracker (Mantis), a server with OpenEMR, a server with a WebShop based on OsCommerce as well as a link to download a version of FreeMind.

When the conference center was getting near closing hours we rounded up and identified tasks needed to be done for the next coming days. We were assigned to setup and fill both the OpenEMR and the WebShop with new test data. We determined that we first should create a data model in FreeMind; both for reviewing before inserting the data and also so that we could print out the model and use it as a means to discuss with participants in the TestLab. With the model at hand, we thought the participants would also be able to identify test ideas of their own that they could try out.

We packed up everything and took Henriks car together with James and Bart. Half an hour later we were down in the hotel bar for dinner. We first met up with Steve Öberg, Rikard Edgren, Björn Karlsson, and Stefan Thoresson; and later on Markus Gärtner, Neil Thompson & Michael Bolton and some more joined our table and we were discussing testing long into the night.

Tuesday

At 8.45 we met up with James and Bart at the hotel and took a cab to the conference center where we setup the TestLab again. We started on the data models while James and Bart worked on setting up the details of the two networks. Meanwhile the sponsors were installing their applications on the client machines. Late into the afternoon we were starting to get ready with the new data for both systems. During the afternoon several people were visiting us and some of them started testing the applications. In fact, the first bug entered in the bug system was the one that won the prize “Best bug” – Markus Gärtner found it pretty fast…
Around 18:00 we were able to move into the room where the TestLab should be. We moved in all servers, routers and clients and setup the power for it all. Funny thing is that we didn’t find more than one electricity socket, so all machines were hooked up on this only one and we decided to load test the electricity over night.
During the evening we attended the Rebel Alliance (or Danish Alliance) for 11 interesting lightning talks. The discussions went on after that long into the night. But to our defense we must say that we got in bed in time.

Wednesday

Despite that we got in bed fairly early, we both overslept and missed the grand opening of the TestLab… Such a good start!
But no time to cry over that; we dug in and assisted in welcoming and introducing people that entered the TestLab and told them about it and how to use it; as well as guiding them to start testing. Bugs were reported and the sound of the chicken echoed all over. Nice!

During the day a lot of people were in and really sat down testing. There were also a lot of interesting collaboration going on between people that hadn’t met before. Some of the people that spent more time than others were Shmuel Gershon, Markus Gärtner, Isabel Evans, Michael Bolton, Ajay Balamurugadas, Zeger Van Hese, and Rob Sabourin (we might have forgot some of the names, but you know who you are).

During the day there were a lot of non-arranged  testing going on. But we had two really appreciated sessions during the day.
Firstly, there was a pair-testing session led by James Lyndsay where close to thirty people attended. There was a good mixture of experienced and less experienced testers. Even though the session was only about 30 minutes divided in three 10 minutes mini-sessions, it seemed to give the participants some very nice hands-on experience to bring home. And a lot of good collaboration was happening!

Secondly, during lunch hour it was an improvised weekend-testing session, Ajay Balamurugadas had invited as many as possible during his talk on Weekend Testing right before the session. Ajay and Markus Gärtner led the session with the rest of the participants that had showed up. Since there were so many that showed up we teamed up in pairs or smaller groups. The collaboration in the TestLab was yet again really great! After close to 30 minutes session we had found a bunch of interesting bugs while at the same time many have had the opportunity to team up with these fabulous  testers.

Despite that we had to cancel some planned sessions, we all thought that the first day was a real success . And we measured that by realizing that we didn’t have had time to take breaks or eat a proper lunch because the TestLab wasn’t empty for even a minute; there were at least 5 people there at minimum (except during the key note by Stuart Reid).

On the evening there was the Gala Awards dinner at Copenhagen City Hall and we ended up at a pub called Bryggeriet with the Rebel Alliance discussing testing in all forms. We went home pretty early and met up with some people in the hotel and continue discussing testing.

Thursday

The day kicked off at 08:00 in the TestLab and we used the first 30 minutes preparing for yet another interesting day!

At the AllSTAR testers, a few renown testers were able to join in. We split the TestLab into three groups where each focused on a specific area which they knew well. Each group attacked the systems in various ways, teaching the participants new ideas on how to test the systems. Ari Takanen was showing how he did some security testing (and/or fuzz testing); Rob Lambert showed how he did accessibility testing; and James Lyndsay had a exploratory testing session discussing a lot of test ideas with the participants.

At the round the table of managing ET, James Lyndsay moderated the group consisting of Rob Sabourin, Michael Bolton, Carsten Feilberg, Shmuel Gershon, Zeger Van Hese, Henrik Andersson, Henrik Emilsson, Martin Jansson, Markus Gärtner, John Stevenson, Neil Thompson and some more (we are sorry if we missed to mention someone, but we try our best to remember all people that were there out of those 200 TestLab attendees). We talked about several aspects around managing ET; and the discussion began with how we had managed our own time in the TestLab. James Lyndsay moderated the group in an excellent way. One of the subjects was how often you change/update your charters. Another interesting topic was on how much information you should provide different testers with.

During the lunch session, Markus Gärtner launched a Testing Dojo that was very appreciated. And Michael Bolton was doing a transcription of the session. The testing dojo meant that one tester tested the application with a projector that showed the computer screen. The tester then described through the testing and explaining what happened. All other around the tester could ask questions. The tester should be switched every 5 minutes letting several people show off their skills.
While this was the intention, the testing dojo slightly turned out to something different. But still as much appreciated by those involved.

Thursday evening we, Markus Gärtner, and Michael Bolton headed in to Copenhagen for a dinner with Ajay Balamurugadas, Teemu Vesala, Kalle Huttunen and Mika Hänninen. It took us two hours to get to the restaurant because we were forced to explore the Copenhagen Metro and train system…

Friday

We headed home to Sweden in the morning and arrived in Gothenburg at 15:00 in the afternoon. Then Henrik continued to Karlstad and arrived at 18:30. Seeing the problems other people had with getting out of Denmark by plane or train, we were lucky that car traffic wasn’t too affected by the blizzard…

Some numbers

• 180 people got in to the TestLab and got introduced to what we do (most of them sat down and tested)
• 100 hours of testing
• 100 reported bugs (plus several bugs not reported in the bug system)

Reflections

Considering how the TestLab was run, it was like a mix of having an open house and being a test lead handling several projects at the same time. We served different stakeholders such as the participants in the lab, the speakers, the conference personal and ourselves as apprentices as well as the masters (James and Bart). The plan is ever changing. Each stakeholder value different things. Just like in the real world outside the TestLab.

There were also 5 sponsor presentations from the tool vendors that had their tools on the client machines. We really appreciated those sponsors that had put in some time to really show how their tool could be used on the same applications as we tested in the TestLab. Very useful and meant that there were more value for those interested in the tools to try things out for themselves on the lab machines.

During the TestLab we got a lot of ideas on what we want to do next year. A lot of ideas that we had on beforehand were though only good in theory; seeing the real obstacles and problems in practice was a good lesson.

And finally, we really appreciate the great effort that James and Bart have done in creating the EuroSTAR TestLab and made it to such a remarkable success. We will have the benefit of building upon a very solid and well thought-out platform & concept!

We had a lot of fun and hope that all others had a great time!
See you all in Manchester 2011!

Cheers,
Henrik and Martin

Last week, me and Martin won the competition “The EuroSTAR Test Lab Apprentices”!
Read more at: http://www.eurostarconferences.com/delegates/the-test-lab-apprentice.aspx

See you in the Test Lab in Copenhagen!

Cheers,
Henrik & Martin

Well, to many people this is nothing new. But still, there are a lot of testers, and indeed test leads, that still think that Exploratory Testing is a technique that can be used in testing. To some extent, it has to do with that both Cem Kaner and James Bach have used this term amongst other techniques (e.g., in the BBST course material). But they have changed and updated presentations as much as possible over the last period of time.

According to Cem Kaner nowadays, the definition of exploratory testing is “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

And this is important. You can come a long way on reaching the style of Exploratory Testing just by treating testers as intelligent people; which is one of the most important factors in the definition above. In contrast to Exploratory Testing you have Scripted Testing that, in my opinion, treats testers as dumb people or even dumb machines. I think that this approach is devastating for our profession (even though I can somehow see the need for Scripted Testing in some places).

A technique is a recipe for solving a problem, whereas a style (or approach) is a way of thinking around a theme that stretches far beyond solving a particular problem.
So when we talk about selling in Exploratory Testing to managers or project stakeholders it is not a technique that we are selling; it is rather an acceptance of a mindset where testers are treated as professional and intellectual human beings that are able to perform Sapient Testing, and particularly in an Exploratory way. It is not about stakeholders investing in a technique, it is about them showing that they have as much trust in testers as they have in other intelligent co-workers of the project.

It was Stockholm again this year. Good to not have to travel far, but since you are travelling I wouldn’t object to something more exotic, and warmer. Next year it is Copenhagen, again.
I had a full-packed program with 4 days of tutorials, workshops, tracks, short talks, test-labbing, conversations, so in total it is quite an amount of ideas since new things appear when you combine what you hear with your won reality and thoughts on testing. I am a bit exhausted after all this, which is a very good thing!

Monday gave Exploratory Testing Masterclass with Michael Bolton. Even though I would have expected a bit more advanced level, it was very good; here are some highlights:
it’s the scripted guys that sit and play with the computer; most of what we look for is implicit, it is tacit; develop a suspicious nature, and a wild imagination; checks are change detectors; some things are so important that you don’t have to write them down; codingqa.com episode 28 is important (I listened to it, but didn’t understand what was so important); managers fear that Exploratory Testing depends on skill, is unstructured, unmanageable, unaccountable; we need to build a “management case” (or should it be middle-management case?)
Michael showed an improved Boundary Value Analysis for a more complex example, where there are many boundaries; whatever focus for coverage you have, you will get other coverage for free; visualizing Test Coverage with sticky notes on a model is a good way of creating charters for Session-Based Test Management.
Go beyond use cases, create rich scenarios; emphasize doing, relax planning; Test Coverage Outline and Risk List to guide future sessions; don’t try to find bugs in the beginning, it takes time away from building a model; HICCUPPS + F (Familiar Problems); you learn best when you are in control of the learning process (and have fun); who said something valuable should be that easy?
Reports to make number people happy; SBTM debriefs are important for keeping quality of the testing and the report (and good for coaching and mentoring); the principal interrupter of testing work is bugs; Weinberg: “everything is information”; Dr. Phil: “How’s that working for you?”
He also had good exercises, and a nice movie, a Detective Story.
I haven’t been to Michael’s tutorials before, so it was about time.

Tuesday started with Tim Koomen tutorial “From Strategy to Techniques”; there’s a gap between the test strategy and the actual tests.
He is very knowledgeable, and walked through the basic testing techniques that every tester should have in his toolbox: Equivalence Partitioning, Boundary Value Analysis, Classification Tree Method, Pairwise, Path Coverage, Condition/Decision Coverage, Input/Output Validation, CRUD, Operational profiles, Load profiles, Right/Fault paths, Checklist.
The examples are focused on functionality, and a magazine discount example is shallow; it doesn’t consider if the person is just about to become 20 or 65 years old, or if you don’t know the age, or if an incorrect age is corrected. And now we haven’t even considered everything else that interacts with this small piece of functionality.
Every time I see this list, I think that they don’t sum up to the testing techniques I actually use when I design tests.
So my highlight was this feeling combined with my shallow knowledge about Grounded Theory; maybe we could have a super-advanced error guessing test technique, that describes the really, really good test design that happens all over the world, where we are looking at a lot more things than the requirements (more to come on this…)
Tim showed PICT tool (consider 1-wise!), and audience mentioned that Mercedes-Benz also has a free tool (see pairwise.org for a long list of tools)
I learned a new thing: the modified conditional coverage, where you omit tests that aren’t likely to catch errors.
Sometimes I wonder how many of the tests from the classic test techniques that preferably are automated in unit tests.

The actual conference started with Lee Copeland talking about nine innovations you should know about: Context-Driven School (the search of best practices is a waste of time); Test-Driven-Development (help you write clean code); Really Good Books (too few testers read the books!); Open Source Tools; Testing Workshops (Specialized focus, participatory style); Freedom of the press (He is no fan of twitter, but like blogs); Virtualization (Rapid setup, state capture, reduced cost); Testing in the Cloud (rent an awful amount of machines, very cheap); Crowdsourced Testing (Lee did not mention the ethical payment dilemma)
“sincerity is the key – once you learn to fake it…”
Keys for future innovation: creative, talented, fearless, visionary, empowered, passionate, multiple disciplines. Do we have all of these???

Johan Jonasson explained Exploratory Testing and Session-Based Test Management, but since this was a short track, there wasn’t so much time left for the real juice. “ET has specific, trainable skills” (Bolton)
Julian Harty, Google (where the testers seem to have huge responsibility areas) explained the concept of Trinity Testing, 30-90 minutes walkthrough(s) by Developer, Tester, Domain Expert. Not radically new, but it felt very fresh and effective. Julian was the only one I saw that brought a hand-out, one paper with the essentials.
Geoff Thompson talked about reporting, that “it’s the job of the communicator to communicate.” 1/10 of men (1/50 for women) are color-blind, and maybe you want everyone to understand the report? (I saw two other presentations, where red-green was used to highlight important differences.) “Know your recipients, what information do they want?”, “honesty is always the best option.”
Michel Bolton had a short session on “Burning Issues of the Day” that is available here. Very funny, very thought-worthy, very good.
Jonathan Kohl talked about Agile having lost a lot of its original value, it is re-branded, old stuff, and has become business. Process focus can distract from skill development, the point is: focus your work on creating value.
I asked Jonathan afterwards about Session Tester (where not much has happened lately), and he said that the programmers are too busy, but it will happen things pretty soon.

Wednesday’s first keynote was Naomi Karten about change; change that represents the loss of control, change that we often respond to in an emotional and visceral way.
Hofstadter’s Law: It always take longer than you expect, even when you take into account Hofstadter’s Law.
Regularly communicate the status of the change, also when you don’t have any news, or when you’re not allowed to tell the news (say that you can’t say anything!)
Listening and empathy are the most important change management tools.
The biggest mistake is to forget the chaos; and in chaos: don’t make any irreversible decisions.
This was my favorite keynote, and as I’m writing this I understand there was some really important information in the presentation.
Mike Ennis talked about software metrics, that help you manage the end game of a software project.
The end game term is taken from chess, where the outcome is almost decided, it is just a matter of technique, primarily about not making mistakes. Mike used the analogy that if you can anticipate what will happen, you know what to do next.
He defined example release criteria, which often aren’t met, but business decisions can overrule the criteria.
40% of the code is about positive requirements, “not a huge fan of exploratory testing, do it if you have time, after the standard tests have been run”.
He used a Spider Chart (aka Radar Plot) to visualize The Big Six Metrics (Test Completion Rate, Test Success Rate, Total Open Defects, Defects Found this week, Code Turmoil, Code Coverage.)
A question was raised that there is a risk of over-simplifying things, and the answer was: “Yes, but these are indicators only.”
Erik Boelen talked about Risk-based test strategy, if you do it with different roles it is like Läkerol, it makes people talk.
He likes games, the we-versus-them game with developers is good, at his place developers with many bugs buy drinks to the testers; and testers aren’t allowed to say one word for a week; last week that word was testing…
A very interesting and nice thing about the presentation was that he explained their (very good, but for some, very provocative, I assume) test method as a natural and obvious way:
They take the entry paths from the Risks and perform Exploratory Testing. For High and Medium risk they document test cases as they explore, and for Low risks they just report the results.
“Eventually testing will rule the world.”
Shrini Kulkarni talked about dangerous metrics, and that software development must consider where it is suitable with measurements. (Shrini hates SMART by the way, so I like him.)
A root cause is that metrics/measurements represent rich multi-dimensional data, there is inevitable information loss.
People might say “we can’t improve without metrics”, but you could use metrics as clues to solve and uncover deeper issues.
We can report with stories attached to the numbers, but still, we are losing information.
Susan Windsor had a double session on communication styles where time flied. In the audience, everyone said No to “Exploratory Testing adds no value”
Art of Storytelling involves: Random, Intuitive, Holistic, Subjective, Looks at wholes (two of my favorite adjectives!)
Research shows that interviewing is the most ineffective method when hiring.
She noted that a high proportion of testers also do creative things like music, poetry (which seems natural, it is good to have trained a lot at being creative.)
We looked at four different Personal Communication Styles (why is it always 4 different types of persons??): Strategist, Mediator, Presenter, Director.
Gitte Ottosen had the ending keynote of the day with a presentation about combining Agile and Maturity Models (“CMM = Consultant Money Maker”)
“Metrics, I know they are dangerous, but also necessary.”
Manual Testing involves using the story to do exploratory testing (“continuous learning as we implement the feature.)

On Thursday I was wise enough to skip 2 sessions in order to have a late breakfast and practice my presentation.
So the first presentation of the day was Zeger van Hese (he won the best paper award for the second time this year) that shared his experiences of introducing Agile, but only doing parts of the full-blown, capital A stuff (resulting in a Real-world, semi-Agile process.)
They used this strange mix of Waterfall and Agile that many, many companies have, and got a better and better situation as more members of the team sat in the same room.
But in the end they fell back to old behavior, there were many late changes, many Release Candidates, and a one month delay. But; excellent quality and stability.
3 Agile goals: better feedback, faster delivery, less waste.
They did a big Agile no-no by using manual testing, which seems like a wise deviation to me.
A quote attributed to Einstein, and several others: “In theory, there is no difference between theory and practice; In practice, there is.”
Next presentation was my favorite of the whole conference: Fiona Charles, Modeling Scenarios with a Framework Based on Data.
They built a conceptual framework at 2 levels: an overall model of the system (testing), and the tests to encapsulate in that model.
They did a structured analysis of all attributes for each framework element, and then used these attributes to build simple, and then more complex, scenarios. This is difficult to do for many testers, so careful review of this work is a way to make sure the results are good.
I think this is an example of the test design technique I thought about on Tuesday, a very advanced, structured way of designing tests that can’t be captured by the classic test design techniques (error-guessing is closest, but there’s a lot more to it.) I like to call this Grounded Test Design (more to come on this…)
“scenario testing is a nice thing to add to your repertoire”, “combine two or more models”, “don’t ever fall in love with your model”; they found 478 bugs, and all except 20 was essential to fix for the customer.
What you need to do something like this: testers with domain experience, business input and scenario review (and maybe an industry book), a model, structured analysis.
After lunch, I had a second session in the Test Lab, so I could report some of the bugs Zeger and I found the day earlier. It was great to test on real stuff, but I didn’t have the time that I would have liked in order to understand the product and its failures. There weren’t time (at least for me) to discuss in depth the findings with other testers, which is something I hope to be able to do next year (I’m hoping the Test Lab will continue.)
At the last presentation slot, I did my thing on “More and Better Test Ideas”. People were tired, but looked interested, so I’m happy with the presentation. I won’t recapitulate the session, but I did talk about the potato, but had to skip the new Find Five Faults analogy (unexpected time pressure, I’m still in doubt that I got the 38 minutes I was supposed to.)
The paper is available here, the presentation here, and it will be given as a EuroSTAR webinar at December 15th.
Good questions, and also examples of how similar approaches are used by others. A bit more than 10% of (almost 100?) attendants use test ideas/conditions.
The next day I got a mail stating that ideas from my presentation could be used at once; the best feedback to hope for.

The Test Lab organizers (James Lyndsay and Bart Knaack) seemed happy when presenting the results, and it’s good to know that the efforts might make open-source medical product OpenEMR a bit better (there is certainly room for improvements…)
At the final panel debate half of the audience voted that certification is important, Tobias Fors shared the insightful “as a developer I was scared about code review, but then I realized it really was about my low self-esteem.”
Regarding teaching testing in school, it was said that critical thinking should be taught early.
“How do we breach the barriers and invite the developers to our world?”
Dorothy Graham (who reviewed every presentation!) ended the conference and announced the next years programme chair John Fodeh.

Overall it was a very nice conference, at the expo Robert from ps_testware was nice and let me win a chess game this year also.
Recurring themes were Agile/Exploratory Testing (why are they grouped together?) and now and then the importance of a Story was emphasized.
Unknown source: “The higher and more complex quality objectives you have, the more manual testing is needed.”
Attending a conference isn’t about learning truths from the experts, it’s more about getting input to be able to create your own ideas that apply to your job, and to meet people, hear stories, interact with people that share your passion: software testing.
See you next year!

/Rikard

I believe that it is impossible to objectively capture important things about a software tester’s productivity.
On the other hand I don’t believe there is a big difference between feeling productive and being productive.

I feel productive when I

* test a feature that is good, but not perfect
* review specifications
* do pair testing
* am happy
* am motivated
* find interesting things in the product
* find very important defects
* report bugs
* help developers
* don’t think much

When do you feel productive?
How do you make sure you spend as much time as possible being/feeling max-productive?

Recently in Sweden there was a tragic death to a young child that could have been rescued if only the child had come to a hospital in time for a full exam. The one that was blamed for this death was the medical care hotline company that did not understand the severity of the illness and did not send this kid to the hospital. (Read more, in Swedish: http://www.dn.se/sthlm/pojke-dog-efter-rad-att-inte-aka-till-sjukhuset-1.859096)

After this tragic accident, it was discovered that this private medical care hotline company pays out a monthly bonus to those nurses that keep their phone calls short. (Read more, in Swedish: http://www.dn.se/sthlm/skoterskor-far-bonus-for-snabba-rad-1.864534 )
I.e. if they keep the call below 3.48 minutes and during that time complete the medical record, they receive a bonus of 1000 Swedish kronor (approx. € 100). In order to receive the bonus, there are some quality goals as well. E.g., you don’t get the bonus if you unnecessarily send someone to the emergency ward; or if you give a faulty medical advice.
Do I need to tell you that the county council paid the private company by the number of calls they handled.

This is what happens when you use simplified and dangerous metrics as a foundation for incentive pay… And these metrics are easy to abuse because they are based on simplified models of how the real world looks like.
When dealing with people, you are dealing with “complex systems” (read more in An Introduction to General Systems Thinking, by Gerald M. Weinberg ) and you cannot treat every person like they would be the same. I.e., the people calling in (and indeed children that cannot speak for themselves) are treated as a neutral “* 1” or “+ 0” in the metrics equation.
This happens if you include simplified metrics to measure your efficiency when dealing with people; metrics that leaves out the most important and complex parts of the equation: humans and human interaction.
Nurses know how to work with people, they know that people are unique; they know that their job is hard and requires skill and years of experience. They know that some patients require 20 minutes before they are calm or they need such time to explain everything important; they also know that some people just need 25 seconds before they are satisfied.
It is a shame that nurses are measured by how fast they finish a phone call.

It is the same thing that happens again and again in software industry; or rather the peopleware industry. People that work with developing software are measured by metrics that are dangerous and wrong; and in many cases it can have the same tragic outcome as with the young boy that did not reach the hospital in time…

Read more about (dangerous) metrics in the Software Industry:
Software Engineering Metrics: What Do They Measure and How Do We Know?, by Cem Kaner.
Metrics, Schmetrics, by Matthew Heusser.
Meaningful Metrics, by Michael Bolton
Measurements/Metrics/Analysis/Judgment, by Rikard Edgren

This was originally a response to Rikard’s post “Multi-Dimensional Software Testing”, but here I have developed my thoughts a bit.

As I see it, there are more or less obvious stakeholders and stakeholders that might be more or less hidden. A “customer” might be such an obvious stakeholder. It might then just be a matter of recognizing which of these “customers” that matter and take those expectations into consideration when selecting test activities.

The project manager is another example of an obvious stakeholder.

And a product owner would naturally be considered an obvious stakeholder.

But how about the hidden stakeholders; and what are their interests in the project/team?

Maybe the members of the test team would have interests in the team’s work that would affect the testing activities (i.e. their own work)? Someone in the team might be dependent on other peoples work. Some people in the team might lack important skills which would affect the chosen test activities. Are there some people that only can work part time and thereby affect which times meetings are held in the team?

The members of the test team are them self stakeholders to the project and might have important interests. This might have impact on the testability, documentation effort, release schedules, communication, etc. Project members other than the testers are also stakeholders to the project and they might have their specific interests.

Business managers might have much to say about the testing activities; and they might especially be interested in the cost of the testing activities. They might have influence on hardware or software that would be needed and thereby affect the testing activities.

Other test groups that work in parallel might be interested in the outcome; and especially to make sure that your team doesn’t over-deliver and make their team look bad. And they might be interested in what type of cooperation that could take place in order to be more effective. They might also have an interest in what type of reporting to do in order to compile a common test report. Other test groups might also be interested in creating a common vocabulary to use in the test scripts.

The maintenance team that would receive the product after it has been released might be interested in the outcome. And especially they might be interested in the format of the test product i.e. documentation, test scripts, procedures, etc. They might also be interested in good practises that have come up during the project; and if these good practises were designed to work for the maintenance team, they have indeed already affected the testing activities in your team.

Certainly there are more hidden stakeholders than those above to take into consideration when selecting test activities or test strategy to focus on. But my goal is to illustrate the problem with a couple of hidden stakeholders in order to make you think about your situation and possible stakeholders that might matter for you.

Note:
Many of these stakeholders are already taken into consideration on a daily basis, but I believe that they are not considered as stakeholders. Maybe you would discover more important aspects of each stakeholder if you tried to analyse each of them more thoroughly.

E.g., try to list all people or roles that matter to the team and start to analyse what their interests are and how they might affect your team/project.

And you should not be surprised that some of these were hidden to you at first, and when their interests are revealed you might discover that they would have a large impact on your testing activities.