As an exercise, try these generic test ideas on any product, for instance a recent upload to sourceforge.net. Then come up with a better test idea, and write as a comment. Also suggest one to remove; I have decided they must be seventeen (a Tranströmer homage)

* Try to do what it is supposed to do
* Verify that Help matches functionality
* Let it run for quite some time, and look at resource utilization
* Play around with core functionality for five minutes
* Ask someone else for their first impression
* Perform any simple scenario as fast as possible
* Use keyboard-only for a while
* Have a look at all installed files
* Get rid of all traces of software (Uninstall)
* Run on a radically different platform, preferably an Error-Prone Machine
* Use credibly dirty data
* Provoke a bunch of failures
* Be a user that want to destroy for others
* Compare with common behavior for the environment
* Review About dialog information
* Is it at least as good as a comparable product?
* Do something valuable for you

Investigate your feelings during these tests; keep usability, charisma, security, performance and testability in the back of your head.
End by describing the state of the product in less than a minute.

These are test ideas you can try on any product, and they will give you interesting information.
However, spending the same amount of time on other tests, might be even better…
Really good test ideas are specific for a product, capturing their essence; challenging a main purpose.

The big drawback and big advantage with reliability testing is that it is easiest and most effective to perform together with other testing. A separate automated reliability regression test suite could cost an awful lot to implement, but reliability in your spine when performing any type of manual test, together with deviations, is cheap, interesting, and powerful.

If you look at Reliability from a standards perspective, you will see a lot of measurement methods like Mean Time Between Failures. You don’t need to use these. You can test and find important information anyway.
The most lightweight method is to ask these questions to heavy users of the product:

Reliability. Does the product work well all the time?

- Stability. Are you experiencing (un)reproducible crashes?
- Robustness. Are there any parts of the product that are fragile and have problems with mis-configurations or corner cases?
- Recoverability. Is it possible/easy to recover after (provoked) fatal errors?
- Resource Usage. What does the CPU, RAM, disk drive et.al. usage look like?
- Data Integrity. Are all sorts of data kept intact in the system?
- Safety. Is it possible to destroy something by (mis)usage of the system?
- Disaster Recovery. What if something really, really bad happens?
- Trustworthiness. Do you feel you can trust the system?

You may also want to perform some specific tests aiming at the different sub-categories.

Stability. Run the product for a long time, without restarts.
Automate a simplistic scenario and run it thousands of times in a sequence.
Count the number of non-reproducible crashes per day that happens to your team.
Try really hard to reproduce the non-reproducible issues.

Robustness. Provoking error messages is fun, and don’t forget to check spelling and if error message helps.
When this is important: hit hard, hit many times.

Recoverability. Turn off the power for machines performing important things, restart and look at behavior.
Whenever an error occurs, try to recover, and consider if it is easy and intuitive.

Resource Usage. Look at system resources now and then.
Stress the system in various ways (but only spend time on this if the project is interested in results.)

Data Integrity. Use all types of data (numeric, strings, out-of-range, invalid, empty, Unicode), in different sizes (small, medium, large) on different systems (localized OS, regional settings, different fonts) through all parts of the system.

Safety. Do thorough brainstorming around scenarios where people can get hurt. Be aware that ambiguous or missing information can be very dangerous if they affect important decisions.

Disaster Recovery. You probably don’t want to test this for real. But you can ask developers or others if there are possibilities of continuing using the software after a crucial machine has disappeared. This is one of those characteristics that either is irrelevant, or very important.

Trustworthiness. Note down all inconsistencies in behavior, or moments when you are unsure what the product is up to.
Tell the project how you feel about the product’s reliability.

The best start to get reliable software is to have really solid code, and properly customized code checker tools can help you with this.

I guess the list could be a bit longer and still be lightweight; feel free to help me out!

When I read about the “classic” test design techniques, I don’t recognize the way I come up with test ideas.
Sure, the implicit equivalence partitioning is used pretty often, and I get happy the few times a state model is appropriate, but the testing I perform seldom has the unit/component focus that these techniques have.
Rather I start with a What If question I believe is important, e.g. what if the Client-Server connection is lost here.
And then I use various sources to find out how this can occur, and write test ideas in an easy-to-read format:

- investigate behavior if Server goes down
- check what happens if network cable is unplugged
- can the function be performed on a really slow network (use NetLimiter)
- check behavior in our load test environment
- what if the connection is interrupted before/in beginning/at the end?
- verify ability to Cancel operation
- are error messages adequately informative? (remember Security, Secrecy…)
- how will a user feel during interruption?
- look at information in the log files

The focus is on the analysis part, to find out what we want to test.
There is a mix of “what” and “how”, non-functional is not a limiting factor, and the details are left for the test executor to work out.
I think this way of designing tests is extremely common, and yet we don’t have a name for it! Or do we?
We should have a name competition.
Vote for one of these; come up with your own suggestion; or refute the whole idea…

* Straightforward Test Design
* Common Sense Testing
* plain English Test Design
* Test Idea Generation
* One-Liner Tests
* Test Design Un-Technique
* Error Guessing
* Test Design
* Risk-based testing
* Analysis-Focused Test Design
* Test Charters
* Zxcvbnm
* Simple Test Design

- So are you promoting a test design technique that anyone can perform?
Yes, I think anyone should be able to write, and understand these test ideas.
To do it good, you just need to understand what is important, see the whole, and realize how it is best achievable in tests.

And if you can write all test ideas with a granularity so all interested parties can get a grip of the whole testing effort, you will get good feedback, and have a great start of your testing story.

It is very difficult to describe the process of synthesizing test ideas. It involves a multitude of information sources, a sense of what’s important, and a dose of creativity to come up with ingenious test ideas, and effective ways to execute them.

The easiest way is to take the requirements document, re-phrase each item, and optionally add some details using equivalence partitioning.

The best way is to use a variety of information sources and generate testworthy test ideas that have a good chance of being effective.
An impossible way is to combine all important information in all possible ways.

Rather you should use each element that is important, and each combination you think is important.

Reviews are helpful, the actual product invites to some tests, and also the speed of the tests will guide you to the best ideas in your situation.

It is recommended to write down the test ideas, at least with a high-level granularity. If reviewing isn’t done, or the software already is available, it is faster to write them afterwards, together with the result.

Don’t try to cover everything, because you can’t. Rather make sure there is a breadth, and count on serendipity to find the information needed for a successful product. Some of the test ideas you will generate on-the-fly, during your test execution when you see what you actually can do with the software.

And don’t stop just because you reach an adequate test idea. Think some more, and you might find better ways, or new solutions to old problems.

I find no better way to elaborate further than to discuss different categories of test ideas:

Ongoing Test Ideas

Ongoing test ideas can’t be completed at once; they keep going as long as more relevant information is revealed. A good example is quality characteristics like stability; the more you test, the more you know about the stability of the product. You probably don’t do only ongoing tests (in the background) for important characteristics, but as a complement, it is very powerful, and very resource efficient.

Another example is ongoing usability testing for free, where the tester keeps the relevant attributes in the back of their head during any testing, and whenever a violation occurs, it is noticed, and communicated.

Classic Test Ideas

Classic test ideas deal with something specific, and can be written in “verify that…” format. They can be a mirror of requirements, or other things the testers know about.

For review and reporting’s sake, beware of granularity; rather use one test “verify appropriate handling of invalid input (empty, string, special ASCII, Unicode, long)” than a dozen of tests. In some situations you can even use “Verify explicit requirements are met”, and put focus on the more interesting test ideas.

Also take the opportunity to recognize which tests are better suited as automated tests, and which are safest to test automatically, and tool-aided and manually.

Combinatorial Test Ideas

Many bugs don’t happen in isolation (that’s why unit tests are very good, but far from a complete solution), therefore testing wants to use features, settings and data together, in different fashions and sequence.

Scenario testing is one way to go, pairwise is said to be good, and the tester with product knowledge can tell you at once which interoperability can’t be neglected.

Combinatorial test ideas use many test elements together, because we think they can have effect on each other, or we don’t know.

Unorthodox Test Ideas

Requirements are often written so they can be “tested” (but what usually is meant is “verified”.) This easily results in requirements being quantified and often in a contractual style, instead of describing the real needs and desires.

The risk of missing what is really important can be mitigated with testing that isn’t necessarily aimed towards falsifying hypothesis. “You are one of few who will examine the full product in detail before it is shipped [Testing Computer Software]

Unorthodox test ideas are open, based on something that is interesting, but without a way to put a Pass/Fail status, and more aimed towards investigation and exploration for useful information.

With creativity you can come up with original ways of solving a testing problem.
Usability testing can be addressed by asking any heavy user what they think.
Charisma might be best evaluated by having a quick look at a dozen of alternative designs.
If there are a hundred options, you might want to take a chance and only look at five random.

These are especially vital to get feedback on, since one unorthodox test idea can generate two others, that are better, or be dismissed as irrelevant.

Visual Test Ideas

Some things can’t be expressed with words. And many things are much more effective to communicate with images. Try to use visual representations of your tests, also because they can generate new thoughts and understandings.

Examples: state models, architectural images, technology representation, test elements relations, inspiring images of any kind

The visual test ideas, as all the others, can be transformed to other types, morphed to many, or what is best suited.

What and How?

Most test ideas will probably be focused on what to test. Some will also state how to test it. This can have many benefits, e.g. if you write you’re going to use Xenu’s Link Checker, and someone points out that there’s another tool that will fit the purpose a lot better.

These categories are artificial, and only used to better explain the processes.
When you know them, forget about them, and re-build your own model of the test design that suit your, your colleagues’ and your company’s thinking. Make sure that you change and add to your test ideas as you learn more.
Your questions, hypothesis and answers will emerge.

It was Stockholm again this year. Good to not have to travel far, but since you are travelling I wouldn’t object to something more exotic, and warmer. Next year it is Copenhagen, again.
I had a full-packed program with 4 days of tutorials, workshops, tracks, short talks, test-labbing, conversations, so in total it is quite an amount of ideas since new things appear when you combine what you hear with your won reality and thoughts on testing. I am a bit exhausted after all this, which is a very good thing!

Monday gave Exploratory Testing Masterclass with Michael Bolton. Even though I would have expected a bit more advanced level, it was very good; here are some highlights:
it’s the scripted guys that sit and play with the computer; most of what we look for is implicit, it is tacit; develop a suspicious nature, and a wild imagination; checks are change detectors; some things are so important that you don’t have to write them down; codingqa.com episode 28 is important (I listened to it, but didn’t understand what was so important); managers fear that Exploratory Testing depends on skill, is unstructured, unmanageable, unaccountable; we need to build a “management case” (or should it be middle-management case?)
Michael showed an improved Boundary Value Analysis for a more complex example, where there are many boundaries; whatever focus for coverage you have, you will get other coverage for free; visualizing Test Coverage with sticky notes on a model is a good way of creating charters for Session-Based Test Management.
Go beyond use cases, create rich scenarios; emphasize doing, relax planning; Test Coverage Outline and Risk List to guide future sessions; don’t try to find bugs in the beginning, it takes time away from building a model; HICCUPPS + F (Familiar Problems); you learn best when you are in control of the learning process (and have fun); who said something valuable should be that easy?
Reports to make number people happy; SBTM debriefs are important for keeping quality of the testing and the report (and good for coaching and mentoring); the principal interrupter of testing work is bugs; Weinberg: “everything is information”; Dr. Phil: “How’s that working for you?”
He also had good exercises, and a nice movie, a Detective Story.
I haven’t been to Michael’s tutorials before, so it was about time.

Tuesday started with Tim Koomen tutorial “From Strategy to Techniques”; there’s a gap between the test strategy and the actual tests.
He is very knowledgeable, and walked through the basic testing techniques that every tester should have in his toolbox: Equivalence Partitioning, Boundary Value Analysis, Classification Tree Method, Pairwise, Path Coverage, Condition/Decision Coverage, Input/Output Validation, CRUD, Operational profiles, Load profiles, Right/Fault paths, Checklist.
The examples are focused on functionality, and a magazine discount example is shallow; it doesn’t consider if the person is just about to become 20 or 65 years old, or if you don’t know the age, or if an incorrect age is corrected. And now we haven’t even considered everything else that interacts with this small piece of functionality.
Every time I see this list, I think that they don’t sum up to the testing techniques I actually use when I design tests.
So my highlight was this feeling combined with my shallow knowledge about Grounded Theory; maybe we could have a super-advanced error guessing test technique, that describes the really, really good test design that happens all over the world, where we are looking at a lot more things than the requirements (more to come on this…)
Tim showed PICT tool (consider 1-wise!), and audience mentioned that Mercedes-Benz also has a free tool (see pairwise.org for a long list of tools)
I learned a new thing: the modified conditional coverage, where you omit tests that aren’t likely to catch errors.
Sometimes I wonder how many of the tests from the classic test techniques that preferably are automated in unit tests.

The actual conference started with Lee Copeland talking about nine innovations you should know about: Context-Driven School (the search of best practices is a waste of time); Test-Driven-Development (help you write clean code); Really Good Books (too few testers read the books!); Open Source Tools; Testing Workshops (Specialized focus, participatory style); Freedom of the press (He is no fan of twitter, but like blogs); Virtualization (Rapid setup, state capture, reduced cost); Testing in the Cloud (rent an awful amount of machines, very cheap); Crowdsourced Testing (Lee did not mention the ethical payment dilemma)
“sincerity is the key – once you learn to fake it…”
Keys for future innovation: creative, talented, fearless, visionary, empowered, passionate, multiple disciplines. Do we have all of these???

Johan Jonasson explained Exploratory Testing and Session-Based Test Management, but since this was a short track, there wasn’t so much time left for the real juice. “ET has specific, trainable skills” (Bolton)
Julian Harty, Google (where the testers seem to have huge responsibility areas) explained the concept of Trinity Testing, 30-90 minutes walkthrough(s) by Developer, Tester, Domain Expert. Not radically new, but it felt very fresh and effective. Julian was the only one I saw that brought a hand-out, one paper with the essentials.
Geoff Thompson talked about reporting, that “it’s the job of the communicator to communicate.” 1/10 of men (1/50 for women) are color-blind, and maybe you want everyone to understand the report? (I saw two other presentations, where red-green was used to highlight important differences.) “Know your recipients, what information do they want?”, “honesty is always the best option.”
Michel Bolton had a short session on “Burning Issues of the Day” that is available here. Very funny, very thought-worthy, very good.
Jonathan Kohl talked about Agile having lost a lot of its original value, it is re-branded, old stuff, and has become business. Process focus can distract from skill development, the point is: focus your work on creating value.
I asked Jonathan afterwards about Session Tester (where not much has happened lately), and he said that the programmers are too busy, but it will happen things pretty soon.

Wednesday’s first keynote was Naomi Karten about change; change that represents the loss of control, change that we often respond to in an emotional and visceral way.
Hofstadter’s Law: It always take longer than you expect, even when you take into account Hofstadter’s Law.
Regularly communicate the status of the change, also when you don’t have any news, or when you’re not allowed to tell the news (say that you can’t say anything!)
Listening and empathy are the most important change management tools.
The biggest mistake is to forget the chaos; and in chaos: don’t make any irreversible decisions.
This was my favorite keynote, and as I’m writing this I understand there was some really important information in the presentation.
Mike Ennis talked about software metrics, that help you manage the end game of a software project.
The end game term is taken from chess, where the outcome is almost decided, it is just a matter of technique, primarily about not making mistakes. Mike used the analogy that if you can anticipate what will happen, you know what to do next.
He defined example release criteria, which often aren’t met, but business decisions can overrule the criteria.
40% of the code is about positive requirements, “not a huge fan of exploratory testing, do it if you have time, after the standard tests have been run”.
He used a Spider Chart (aka Radar Plot) to visualize The Big Six Metrics (Test Completion Rate, Test Success Rate, Total Open Defects, Defects Found this week, Code Turmoil, Code Coverage.)
A question was raised that there is a risk of over-simplifying things, and the answer was: “Yes, but these are indicators only.”
Erik Boelen talked about Risk-based test strategy, if you do it with different roles it is like Läkerol, it makes people talk.
He likes games, the we-versus-them game with developers is good, at his place developers with many bugs buy drinks to the testers; and testers aren’t allowed to say one word for a week; last week that word was testing…
A very interesting and nice thing about the presentation was that he explained their (very good, but for some, very provocative, I assume) test method as a natural and obvious way:
They take the entry paths from the Risks and perform Exploratory Testing. For High and Medium risk they document test cases as they explore, and for Low risks they just report the results.
“Eventually testing will rule the world.”
Shrini Kulkarni talked about dangerous metrics, and that software development must consider where it is suitable with measurements. (Shrini hates SMART by the way, so I like him.)
A root cause is that metrics/measurements represent rich multi-dimensional data, there is inevitable information loss.
People might say “we can’t improve without metrics”, but you could use metrics as clues to solve and uncover deeper issues.
We can report with stories attached to the numbers, but still, we are losing information.
Susan Windsor had a double session on communication styles where time flied. In the audience, everyone said No to “Exploratory Testing adds no value”
Art of Storytelling involves: Random, Intuitive, Holistic, Subjective, Looks at wholes (two of my favorite adjectives!)
Research shows that interviewing is the most ineffective method when hiring.
She noted that a high proportion of testers also do creative things like music, poetry (which seems natural, it is good to have trained a lot at being creative.)
We looked at four different Personal Communication Styles (why is it always 4 different types of persons??): Strategist, Mediator, Presenter, Director.
Gitte Ottosen had the ending keynote of the day with a presentation about combining Agile and Maturity Models (“CMM = Consultant Money Maker”)
“Metrics, I know they are dangerous, but also necessary.”
Manual Testing involves using the story to do exploratory testing (“continuous learning as we implement the feature.)

On Thursday I was wise enough to skip 2 sessions in order to have a late breakfast and practice my presentation.
So the first presentation of the day was Zeger van Hese (he won the best paper award for the second time this year) that shared his experiences of introducing Agile, but only doing parts of the full-blown, capital A stuff (resulting in a Real-world, semi-Agile process.)
They used this strange mix of Waterfall and Agile that many, many companies have, and got a better and better situation as more members of the team sat in the same room.
But in the end they fell back to old behavior, there were many late changes, many Release Candidates, and a one month delay. But; excellent quality and stability.
3 Agile goals: better feedback, faster delivery, less waste.
They did a big Agile no-no by using manual testing, which seems like a wise deviation to me.
A quote attributed to Einstein, and several others: “In theory, there is no difference between theory and practice; In practice, there is.”
Next presentation was my favorite of the whole conference: Fiona Charles, Modeling Scenarios with a Framework Based on Data.
They built a conceptual framework at 2 levels: an overall model of the system (testing), and the tests to encapsulate in that model.
They did a structured analysis of all attributes for each framework element, and then used these attributes to build simple, and then more complex, scenarios. This is difficult to do for many testers, so careful review of this work is a way to make sure the results are good.
I think this is an example of the test design technique I thought about on Tuesday, a very advanced, structured way of designing tests that can’t be captured by the classic test design techniques (error-guessing is closest, but there’s a lot more to it.) I like to call this Grounded Test Design (more to come on this…)
“scenario testing is a nice thing to add to your repertoire”, “combine two or more models”, “don’t ever fall in love with your model”; they found 478 bugs, and all except 20 was essential to fix for the customer.
What you need to do something like this: testers with domain experience, business input and scenario review (and maybe an industry book), a model, structured analysis.
After lunch, I had a second session in the Test Lab, so I could report some of the bugs Zeger and I found the day earlier. It was great to test on real stuff, but I didn’t have the time that I would have liked in order to understand the product and its failures. There weren’t time (at least for me) to discuss in depth the findings with other testers, which is something I hope to be able to do next year (I’m hoping the Test Lab will continue.)
At the last presentation slot, I did my thing on “More and Better Test Ideas”. People were tired, but looked interested, so I’m happy with the presentation. I won’t recapitulate the session, but I did talk about the potato, but had to skip the new Find Five Faults analogy (unexpected time pressure, I’m still in doubt that I got the 38 minutes I was supposed to.)
The paper is available here, the presentation here, and it will be given as a EuroSTAR webinar at December 15th.
Good questions, and also examples of how similar approaches are used by others. A bit more than 10% of (almost 100?) attendants use test ideas/conditions.
The next day I got a mail stating that ideas from my presentation could be used at once; the best feedback to hope for.

The Test Lab organizers (James Lyndsay and Bart Knaack) seemed happy when presenting the results, and it’s good to know that the efforts might make open-source medical product OpenEMR a bit better (there is certainly room for improvements…)
At the final panel debate half of the audience voted that certification is important, Tobias Fors shared the insightful “as a developer I was scared about code review, but then I realized it really was about my low self-esteem.”
Regarding teaching testing in school, it was said that critical thinking should be taught early.
“How do we breach the barriers and invite the developers to our world?”
Dorothy Graham (who reviewed every presentation!) ended the conference and announced the next years programme chair John Fodeh.

Overall it was a very nice conference, at the expo Robert from ps_testware was nice and let me win a chess game this year also.
Recurring themes were Agile/Exploratory Testing (why are they grouped together?) and now and then the importance of a Story was emphasized.
Unknown source: “The higher and more complex quality objectives you have, the more manual testing is needed.”
Attending a conference isn’t about learning truths from the experts, it’s more about getting input to be able to create your own ideas that apply to your job, and to meet people, hear stories, interact with people that share your passion: software testing.
See you next year!

/Rikard

In order to become a successful inquisitive tester, there are a couple of things you can do to improve your skills beyond the more common quest to “question a product”. One important thing is to question the tests themselves.

——————–

Have you ever run tests and wondered if they were really necessary, perhaps knowing that the tests are useless?
Have you ever run tests that were too old and not updated with latest terminology or functionality?
Do your tests contain too much configuration information? Should this be put elsewhere?
Have tests become redundant because those failures no longer happen, ever?
Have the intent of the test been lost and therefore been rendered useless?
Have the tests already been run, on the same build? Twice?
Have you found any bugs or any important information at all with the tests?

Are your tests the most powerful?
Are the tests credible?
Can the tests be faster to execute?
Can you run the most important tests first?
Are the tests too narrow and/or too general?
Do you really understand the test?
Have the original test idea been lost in translation?
Are the tests too much of a projection of the test designer’s thought of view?

Are your tests interesting or boring to execute?
Are the tests in line with your test strategies?
How often do you change your test approaches?
Can the tests instead be better used as input to developers’ unit tests?
Can the essence of the tests be used elsewhere?
Have your tests been reviewed by your colleagues, including technical writers?
Have your scenario tests been reviewed by business people?
Have you captured how the users will use the software in the tests?

Are you satisfied with your tests?
Are your (hidden) stakeholders satisfied with your tests?

——————–

Can you come up with more questions?
Regards,
the test eye (Henrik, Martin & Rikard)

When you come up with a new test idea, you are using your knowledge and experience, but there is also some sort of stimuli that triggers the idea. Something you see, hear, understand or think about.

You seldom think in totally new ways, you rather combine things in a new way.

These are my favorite test idea triggers:

* think about each feature (and connect with testing or technical knowledge)

* create or understand a model of the software

* think about interaction between items in the system

* thinking about quality attributes (CRUSSPIC STMPL + Accessibility)

* read bug report titles for similar functionality

* read test idea lists

* play around with the software, or a prototype, or a previous version

* change perspective, see your software as a little box among many others

What triggers you?

Do you get help from Session Tester Primers?
(http://sessiontester.openqa.org/download.html)

At EuroSTAR 2009 I will present “More and Better Test Ideas“; the main idea being that testers could generate many different types of test ideas, and communicate them in a condensed one-liner format.
If you have great tips on how to come up with really good test ideas, or want to review the paper I’m about to write, let me know.

/Rikard

Sometimes it is said that each test case must have an expected result, or even worse, that each step of a test case must have an expected result.
This is the extreme of scripted testing that I dislike for two reasons:
* It takes a lot of time to write and follow detailed test cases; time that is better spent testing things you haven’t had the time or enough information to thoroughly document.
* It takes away the freedom of the tester to try things a bit different, so we will learn less, narrow our thinking, have less chance of finding new, interesting information, and it will be boring to test.

I understand that the reason for the details is that you want to be sure exactly what has been tested.
But the problem with this reasoning is that in software testing, there are so many things that aren’t tested. You are not testing on the exact same data that your customers use, you are not testing on the exact same hardware and interacting software, and you are not testing in the exact same way as the customers will use your software.
So we can be sure which tests have been run, but that doesn’t necesarily mean it will work for the customers.

This is an example where (Bad) Process is way too much more important than Content.

I prefer a lot of one-liner test ideas, that either can be used in vague test cases or as starting point for Exploratory testing.