On 7-9 May the Let’s Test Conference will take place. During the day there will be lots of interesting tutorials, keynotes and sessions. During the evening the events will continue. One of these activities is the Testlab, that we call Let’s TestLab. Initially I misunderstood Henrik Emilsson when we started to organise the lab. I thought the evening event was the testlab. At the time I did not consider anything wrong with having 150 people or more in the testlab. As I saw the evening event program I considered how could I compete with such a fine setup of activities. Well, this is a conference with many context-driven people. We will have different interests, expectations and focus area. So instead of admiting defeat, I considered what elements that we could add to the testlab to bring a great crowd, but perhaps not all of them.

Here is the line up of events in the testlab:

• Collaborative test planning
• Group Test Experiment
• Test Competition
• Making a compelling status report

Collaborative test planning

We will create possible charters, missions for the coming testing in the lab so that those who wish can practise different testing techniques. Everyone is invited to share his or her idea on how to plan testing in through collaboration.

Part of the line up in this event is:

• Rikard Edgren
• Christin Wiedemann

Group Test Experiment

The context of the testlab will be single testers or groups of testers going in and out of the testlab in different time intervals. Each tester will be unique in the sense that they bring different level of experience, skills and approaches to testing. Based on this, we will start experimenting with group testing. I think we have limited ourselves too long with the setup of pair testing. Going back to the early recommendations and experiences from Brian Marick, James Bach, Cem Kaner, Jonathan Kohl and many others, the setup is nearly the same. With new tools and techniques appearing over the years, some assumptions could be questioned.

Let us assume that there are different aspects and combinations of group testing that serves different purposes, we can say that there are different dimensions that could be explored.

Here are a few dimensions that we will experimented with:

• How many testers (2 or more)
• What role you play as a tester
• User types, User Scenarios or storytelling
• Mission of group test
• Note taking techniques
• Partner combination
• Lateral thinking aspects
• Personality types
• Debriefing techniques
• Accountability
• Focus areas or Characteristic focus
• Test Environment
• Basic Configuration Matrix
• … and new ones that we find along the way

We will explore and experiment with different tools that we use when group testing and share experience on what works in what context. We also experiment with a few pre-defined group test setups such as:

• Cinema testing
• 15-min test run
• Coaching a group of testers
• Wolf pack concept
• Testing Dojo

Part of the line up in the test lab is:

• Ann-Marie Charrett
• Markus Gärtner and Meke Mertsch
• Johan Åtting

Test Competition

Can you really compete in testing? Can you compete between two or more teams?  Can you really estimate the value of one piece of information against another? Well, it depends.

I have been an Ultimate Frisbee player for 34 years. I’ve not played for some time, but once a frisbee fan, always a fan. I think the same goes for testing. There are many things that I feel is similar. Craftmanship/sportmanship and passion is major part. Here is one description I like:

Ultimate has traditionally relied upon a spirit of sportsmanship which places the responsibility for fair play on the player. Highly competitive play is encouraged, but never at the expense of the bond of mutual respect between players, adherence to the agreed upon rules of the game, or the basic joy of play. Protection of these vital elements serves to eliminate adverse conduct from the Ultimate field. Such actions as taunting of opposing players, dangerous aggression, intentional fouling, or other “win-at-all-costs” behavior are contrary to the spirit of the game and must be avoided by all players.

One important element in Ultimate Frisbee is the spirit of the game, which you can see more in detail here [1]. Passion and humility as a tester are important traits, the spirit of the game concept might help us here.

So, my idea for a tester competition will be based on some of these ideas. Two teams compete against each other in form of best bugs and session notes. The two teams go through the opposite teams material and conclude who they think should be the winner with a good reason why.

Make a compelling status report

As the last event in the testlab we want to investigate how we can make a compelling status report for our stakeholders. Having many different testers, session notes spread all over, half-finished bug reports and test ideas half-finished… can we create something that is still valuable to someone? I guess this is a common situation at any lab at any company, still we will dig deep into how to go about this.

Part of the line up in the test lab is:

•  Ben Kelly

References

[1] Ultimate Frisbee Rules –  http://www.cs.rochester.edu/u/ferguson/ultimate/ultimate-simple.html

Many of us has a title that is connected to what we do at work. Every now and then I come across titles that makes me wonder what it really means. This time it is one that has been around for some time now: Thought Lead, what does this mean? I would not be suprised if there is a good explanation of how this title first came into place, but I do not think this is well known and I have not researched it.

You can learn a lot by testing something very different from your normal job. I’d love to professionally test a suggested law, or a chainsaw. For now, I give you an opportunity to test a bread recipe, in English or Swedish.

FAVORITE SOURDOUGH BREAD

FAVORITBRÖDET

It should be possible to bake from it if you know, or can learn, how to breed a sourdough, and bake any bread. I’m interested in correctness, ease of use, inspiration and taste (the last one only for the bread, not the documentation) and other things you think might be worth knowing about.

If you only have a little time and interest, the challenge is to come up with the different test strategies you would use.

Test results in comments or in mail (About page) are more than welcome!

If performance is crucial for product success, you probably need pretty advanced tools to measure various aspects of your product, to find all bottlenecks and time thiefs. For all other software, performance is just very important, and you might get by with lightweight test methods. You may, or may not have quantified performance requirements, but you should test performance to some degree anyway; for the whole, but also for each detail (when appropriate.)

In TheTestEye’s classification, performance consists of:

Performance: Is the product fast enough?

- Capacity: the many limits of the product, for different circumstances (e.g. slow network.)
- Resource Utilization: appropriate usage of memory, storage and other resources.
- Responsiveness: the speed of which an action is (perceived as) performed.
- Availability: the system is available for use when it should be.
- Throughput: the products ability to process many, many things.
- Endurance: can the product handle load for a long time?
- Feedback: is the feedback from the system on user actions appropriate?
- Scalability: how well does the product scale up, out or down?

Be aware of different definitions of performance testing, e.g. some include reliability, stress handling, robustness, and what stakeholders believe is most important might differ (even when using the same words…)

Ongoing Violation Awareness

The number one lightweight method starts by finding out which of these characteristics that are relevant for your product. Then keep them in the back of your head, and whenever you see something fishy, investigate further and communicate. Often the OK zone is easy to reach, but testers should notice when violations occur. When appropriate, apply the destructive principle: Increase the amount of everything that can be increased.

No Tools

Perceived performance is what matters for end users (but maybe not for a product comparison check list) so think about how it feels, and try using a stop watch. You might get pretty far by load testing with colleagues with several instances each.

Tools

There exists limiters for CPU, RAM, bandwidth etc. and many of them are free (and some of them become obsolete.) A task manager/resource utilization tool can give you hints on memory, CPU, disk, network et.al. Scripting your product to run over weekend is good for endurance and stability testing. JMeter is free and often quick to get running.

Summarizing

Summarizing performance test results is difficult. Aggregations of measurements don’t tell the full story, and the whole story takes a long time to tell. Communicate what is important, which is easier if you have asked stakeholders beforehand.

Warning: For some products, users aren’t as interested in Performance as the developers…

Henrik Emilsson has convinced me that skilled software testing is based on invisible mental models that help us see what can be tested.
If we can make these visible, we can sharpen our skills, and also teach testing more effectively.
Here follows a simple example I used in class, that shows that by switching between several models you will get different test ideas.
If you want to learn by experiencing, download Perfect Age Calculator application, and test for ten minutes or so.

Example 1 – The User Interface

OK, you’re casually reading a testing blog, that’s fine.
This is the User Interface, and the way you see it is your very first model, giving you (at least) “click these” test ideas.

Example 2 – Timeline

I guess you immediately tested some familiar dates, e.g. your own birthday to see if it handles normal data (Benevolent Start Heuristic)
Many of you checked today’s date and a date in the future.
These tests are probably based on your model of time, which can look in many ways, but something like a timeline (which can be very different for people!)

Example 3 – Detailed Timeline

The time model can be more elaborated, e.g. including mental notes of leap years, and other special dates. This, the previous and next models are visualizations of your equivalence partitioning.

Example 4 – Year, Month, Day

Some switched to test ideas bases on the date being three parts: Year, Month, Day.
This (more easily) rendered test ideas like same year, month, date; and invalid entries for each of these three. You were also more alert to problems with calculation of each of these three.

Example 5 -Technical Flow

Some of you used a technical flow model, reversing what the code is doing. Depending on your knowledge of data types this generated test ideas challenging the boundaries, perhaps by using “den 5 februari 2012″ on a Swedish machine.

Example 6 -Quality Characteristics

Some testers always apply a characteristics model, personally I use an instinctive sub-set of thetesteye’s list, which in this case gave test ideas for intuitiveness, professionalism, memory consumption etc.

Using models

There could be variations of these, and many more as well (keyboard/mouse interactions, surroundings, competitors, familiar problems…); and I encourage you to find these in yourself, so you can make even better use of them.
(For a complex product, there will be hundreds of possible models, and a good way to find some of them is to combine your knowledge with SFDPOT from James Bach’s Heuristic Test Strategy Model.)

The results from these tests on Perfect Age Calculator show a lot of problems, seemingly too many for such a simple program.
When this happens, you should question your models (I Might Be Wrong Heuristic); I started to wonder if there might be some other kind of calculation they try to make.
But Sourceforge project information reads: “This tool is very useful for official use like office, schools, institutes etc.”
So my conclusion is that the piece of software doesn’t live up to its claims.

You don’t need to explicitly visualize your many mental models while testing, as a matter of fact, you shouldn’t, because it would take too much time.
Most of you reading this blog probably do it instinctively, and in different ways (hence the diversity testing embraces.)
But you might need to make the models explicit a couple of times, to train your testing brain to think in many different models, to get a richer set of test ideas. And by sharing our invisible models, I think we can get even better at testing.

The even more important question about which of these tests that are important, well that’s another story…

Status reporting of testing activities is extremely project-dependent. The needs of when and how and what will differ every time. Maybe that’s why there’s so little good writing about status communication; you have to make it up every time.

Templates are out of the question, and I believe examples will mislead you as well.
You’re better off by thinking around these questions (inspired by final section in Lessons Learned)

Why are you reporting?

Are you giving quality assessments, trying to prove why testing is taking place?
Is focus on the product story or the testing story?
What kind of information can change decisions that are based on your report?

What is most important?

The everlasting and most difficult question for all testing efforts.
This is what you should start your report with.

To whom are you reporting?

What kind of information is the audience interested in?
Are developers the only ones reading bug reports?
Do you have a grounded quality model?

Are you adjusting the language for the audience?

Communication includes making sure it is understood.

In which ways are you reporting?

Just one way is rarely enough; use talking, writing, long, short, formal, informal.

Are there certain times when things need to be communicated?

Project milestones and meetings might need your information.
Some things must be communicated continuously and timely; bringing up bad news on the last day is a very bad idea (but sometimes inevitable.)

Are you using status reporting as a means for new test ideas?

I often start writing an important report early, to identify things that needs more testing.
You should start thinking about the reporting before the testing starts.

Are you reporting just because you are supposed to?

Then you should read these questions again…

If status reporting is difficult, there might be something wrong with your testing.

- I saw you tested “42″. How come you didn’t try “43″?
- That’s obvious. It would be the same. No need to test something that would give the same result.
- OK, so I guess you are familiar with equivalence partitioning?
- Beg your pardon?

It is quite often said that testers don’t know the basic test design techniques.
This is both true and false.
“All” testers know how to use basic equivalence partitioning, it is common sense: you put things in groups, so you don’t have to test similar things that would yield the same information.
But they don’t know the name, and they don’t know the theory behind partitioning many types of groups.
But if they are testing all important partitions, is it a problem?

Might be, but probably not. I think a tester with knowledge about the product and its usage will make a common sense selection of the partitions that are needed to be tested for that particular situation.
For cases where you need to be very thorough this will include inputs, outputs, combinations; and extensions to examples like these:

Numericals: normal, inside/outside boundaries, min/max data type, different data types, decimals, precision
Strings: A-Z, ÅÄÖ, Unicode, empty, many, extremely many, control chars, leading/trailing spaces, things that mean other things (*, SQL, links, special values)

- You know you can’t be called a real tester if you don’t know how to do equivalence partitioning.
- Be so it. I just test what is important in my project.

Storytelling has been rising for quite some years and it will soon boom for software testing.
The reason is simple: people like stories.
And if it is used as status reporting instead of lame numbers, it is a step in the right direction, to say the least.
But when testing this idea theoretically, I find some fears:
* they will take long time to tell
* they will be too entertaining, less informative

And when comparing with my experiences, I find success stories when remembering conversations where I fast communicated the essence.

We already have a core tester skill in writing effective bug titles.
We should learn how to do this for whole test efforts, we should always have an executive summary up our sleeve.
We shouldn’t tell stories, it should be more like a librarian’s summary.

To accomplish this we need a lot of training, but I doubt it will be enough.
We also need more words.
They don’t have to be unanimously defined, and they can be reconstructed at each company, as long as they are useful.
Do a daily exercise of explaining the state of the product in thirty seconds, and try out your own terminology.

We need more words to effectively communicate the essence of our findings.

James Bach and Michael Bolton has a classic collection of consistency oracles; HICCUPPS(F): History, Image, Comparable Products, Claims, User Expectations, Product, Purpose, Standards and Statutes, Familiarity

It is a very good collection; not only helping you find out if something is a problem or not, but also the other way round: serving as testing inspiration.
At EuroSTAR 2009 tutorial Exploratory Testing Masterclass, Michael said that anyone coming up with additions would become famous.
As a tester this was of course an interesting challenge!
So over time, I now and then have thought of additions, but they haven’t been solid.
For example; V as in Violation of Quality Characteristics, could be seen as an inverted Familiar Problems or User Expectations.
You, as in the tester’s subjectivity can also be part of F and U.

But finally I found the important oracle missing from their list.
C as in Conversations.
You talk to a developer or other stakeholder, and collaborate to reach a decision.
Yes, during the discussion you might use HICCUPPS(F) oracles, but not necessarily. And it might not be the whole story, and not your oracle.
It might not be a 100% consistency heuristic, but is important enough to not leave out.
C as in Consistent with Conclusions from Collaborative Communication.

Outside the list we have the important No Oracle, where judgment is suspended, and noteworthy information is communicated, without deciding if it’s a problem or not.
But that doesn’t make for a nice soccer-sounding mnemonic, and is better left as a side-note to HICCUPPS F.C. heuristic.

(When double-checking, I noticed that they recently have added another one in the RST slides: Explainability – The system is consistent with my ability to explain it.
For now, I’ll put that one as a sub-part of Conversations; in dialogue, we weren’t able to explain the behaviour.)

History - the product should be consistent with previous versions
Image - consistency with the looks and behavior expected of the product
Comparable Products - consistency with competitors, references or other solutions to similar problems
Claims - consistent with written or oral statements about the product
User Expectations - consistent with what a reasonable user would expect
Product - consistent within itself
Purpose - consistent with the reasons for making/using the product
Standards and Statutes – consistent with standards or other authorities
Familiarity - inconsistent with problems seen before
Conversations - consistent with conclusions from dialogue

During my first paternity leave I learned sourdough baking. During the second I couldn’t help writing an ambitious paper, or a small book, about people-oriented test design, about things beyond test design techniques, close to the exploratory testing tradition.

It can be downloaded here.

It contains collections of knowledge, and generalizations of my ten years of testing the same product suite. I think it can be useful for ambitious testers that want to find any problems that might be important.

It probably is too much, theoretical, irrelevant or condense for many of you, but if you want to give it a shot I recommend the following:

* Download The Little Black Book on Test Design
* Print as double-sided A5 Booklet
* Find a quiet, comfortable place
* Read and relate to your test reality

Comments are welcome, especially additions to the collection of one hundred and three test design heuristics.