thoughts from the test eye

If you just do what’s on the list of things to do, I think you can accomplish decent things, but nothing great.
I don’t dare saying this is a general truth for everybody creating something new, so let’s focus on software development.
There are many management models, and many of them boils down to something like “do the things on this prioritized list”, and the list might be items in a detailed project plan, a Backlog, test cases or sticky notes.
As an attack, and a defense, of these systems I’m saying:

to accomplish something great you got to do a lot more than what is on the list

The reason for this is that you cannot in advance know about all important things and possibilities that are associated with what your team is trying to accomplish.
Great software has even more than the right capabilities, impeccable reliability, super-performance and perfect usability.
There are no Gods in software development that can specify the details in advance.
And if there was an almighty capable of doing this, it would take too much time to read and understand all items on the list.

If the input for the software is taken from the actual demands of the customers, you can give them what they want, but not necessarily what they need, and probably not something that will be very valuable for many customers. And the item on the list will probably focus on functionality, without all implicit details around quality characteristics that can sum up to a great product.

So the solution is to take height for this, for example by
* not performing (detailed) time estimation and planning
* allocate time for “creative slack”
* pad all estimates with an appropriate percentage
* add extra time for polishing (that must be spread out from start to finish)
* rotate free, unscheduled roles
* encourage everybody to pursue interesting things that emerge
* perform one unscheduled “good” thing for every other item on the list
* schedule vague items that can mean almost anything
* but most importantly: have skilled people doing their best to create valuable software

Usability is an important part of any product (if it’s too difficult to use, it  doesn’t matter how great the functionality is) and thereby an important characteristic for the testing team.

But when reading about usability testing, it often involves an outside person trying to use a feature for the first time. Now this is a good thing, but there is a lot more to Usability than Learnability.
There also exist systematic inspection methods, e.g. Heuristic Evaluation by Jacob Nielsen, that should be performed by super-experts on usability.
He recommends alternating between user testing and evaluation, but I think a manual software tester can do both at the same time, if she has (some) knowledge about the domain, (some) knowledge about the product, and (some) knowledge about usability problems.

Here is a very cheap way to do usability testing; ask the following questions to someone with many hours of using the product (probably a real user or a manual system tester)

General usability: What’s your most important usability issues?
Affordance: Would you say that the products invites to discovering possibilities, or are there features you learned too late?
Intuitiveness: Is it easy to understand and explain what the product can do?
Minimalism:Have you seen redundant information anywhere? Any important information missing?
Learnability: Are there any areas you feel are difficult to understand and learn?
Memorability: Do you forget how to perform some actions?
Discoverability: Can you discover all available operations by (systematic) exploration of the user interface?
Operability: Are any common operations cumbersome?
Interactivity: Are there situations where you aren’t sure of which kinds of interactivity is supported?
Errors: Have you encountered strange error messages, or situations that were difficult to recover from?
Consistency: Is the user interface annoyingly different anywhere, or hard to understand?
Tailorability: Are there any (default) settings you would like to change?
Accessibility: Did you notice any problems while using High DPI and mostly the keyboard? Color-blindness?
Localization: Any experiences of running translated product or data?
Documentation: Are there any information you would like to see in the Help?

You might already know about many of the issues in the answers, but now you know more about their importance.
You won’t get any quantitative measurements, but hopefully a lot of valuable information.

Even better might be to have questions like these in the back of your head when executing manual tests, you’ll probably get valuable information for free. There is no need to memorize the categories, the important ones will emerge.

I believe that as a tester you do not value the same things as someone with a different role in the organisation. Each individual most certainly value different things, but I think there are some specific traits of the organisation that triggers a tester to stay or seek new employments.

As a test consultant you value different things. You move into different organisations knowing that you probably won’t stay long, still there are some areas that can be similar to what you value as a more permanent tester.

These are some of the things I value:

Test Approach: I value Exploratory test approach highly. If I enter an organisation with Script-based test approach I try to get it to move toward Exploratory.

Lovable product[s]: If I am to be an assest to the organisation I need to love the product or product family. At least have an interest in making the product something you take pride in.

Other testers: I think you work best as a team in testing. Therefore you co-workers become especially important. Do they have the same keen interest of testing as you? Do they also share the same passion?

Developers: I value the developers a lot. If you are working with good, open-minded, passionate developers it makes everything so much more fun.

Freedom: I value to have the freedom to do and say what I feel is right, even if it might hurt a bit for the organisation.

Trust: If I feel that my superiors and my co-workers trust in what I do and what I say, I will feel more confident and feel that my work brings value.

CEO: If the captain of the ship steers the vessel where I want to go, it makes a big difference for me as a tester. If the CEO think testers are important and that they add value, it is a key part for staying around.

Why did you seek an employment where you are today? When things are getting tough and it becomes hard to understand why you stay, what is keeping you?

What makes you stay?

It’s a truth that we can’t find all bugs, but is it really a truth that we can’t find all important bugs?
And it’s a cliche when used as answer to the (sincere) “why didn’t you find that bug?” question.
Testers are paid to find important information about what they are testing, and included are the big bugs, so you can fix them before the software is used in production.

Besides some good things about forthcoming test strategies, I hope you also can say: “if you have read the test plan and status reports it shouldn’t be a surprise bugs like this got away.” Well-informed risk taking is part of our business.
However, if there is a test that seems important, and would take a couple of minutes to perform, this argument is not valid. Testers (in a serious project) should try to perform all important tests that can be performed fast.

There are testing lessons to be learned if the answer is:
* we didn’t think about that scenario
* it wasn’t included in the requirements
* someone told us not to test that
* we had no idea that the customers where running that kind of configuration
* we found it, but didn’t report it well enough
* we didn’t put the bug there, and they hid it awfully well
* this type of issue can’t be found with automated tests

And there is the occasional invalid question, e.g. when the bug was reported in a good way, or when there was no testing time alotted for a late change.

Bottom line is: take the question seriously, and try to improve your testing to avoid this and other important issues (next time it will probably be another kind of issue…)

It has been said by many, but I heard it from Fiona Charles: “don’t ever fall in love with your model”, and this is a warning I want to elaborate.
A model is a powerful way to understand how the system works, and thereby also how it can fail.
But a model can also narrow your thinking, and stop you from testing outside the obvious patterns, in the ways your customers most certainly will.
So as a tester, you should use multiple (sometimes incoherent and contradictive) models, so you get a better coverage of the potential usage.
Having one model only is more dangerous than having none.

Here are some ideas that might help you use/create more models:

* requirements, and make sure to look for implicit as well as the explicit
* specifications are models, i.e. interpretations of what should be done; you’ll get different test ideas from functional, conceptual, technical and design specification
* the code is also a model, reading all of it is generally not feasible, but there might be some parts that are extra fruitful
* images; drawings, diagrams, mind maps, lists et.al. can show how things fit together, and if you don’t have them, you can create them (but don’t try to put everything in one place, rather use several with different angles, e.g. functional, architectural, technological, scenario-based…)
* test models – the way you think about, categorize and organize your tests, and the elements therein, is a model worth being aware of
* competitors products also model a solution to customers’ problems, and there might exist un-computerized solutions worth looking at
* the product in itself could be the best model; prototypes and early builds generate a lot of test ideas, especially because you can see how things can interact

You should be aware that each tester probably has his own implicit model of the system. This includes different aspects and is influenced by the functionality she has been testing, and which types of issues she has experienced, or heard about. It probably also includes a list of the most important quality characteristics, and it is a silent guide through the daily testing.
An implicit model is difficult to communicate, though.

With many models you can cover the product with different types of tests, and find more important things in the neverending flow of information.

First of all. I am honored to be invited as a guest writer at the testeye! I will start my contribution by asking for your assistance.

In the next year I have been asked to give a couple of classes for test managers that are fairly new in their role. I have been looking through the content description of a class I am supposed to take over and realize that I have a problem. So much of it seems to be picked from an ISTQB-class and just seem to be cementing the old ways of testing. I refuse to spend half a day talking about eight ways of measuring test which means measuring quality and is based on test case success ratio. And I am not going to dive into standards too much either.

So what am I going to put in a two day class for new test managers? These are some of the more important subjects I want to talk about.

1. Schools of testing: a test manager needs to understand that there are some fundamentally different ways that testing can be done. Since many of the students will be working as consultants they need to be prepared for the many different company approaches they may encounter.
2. Test Strategy: What is the testers’ mission. How do I plan my resources and my tasks. Time estimation. What do I need to know to do a good job. What kind of documentation should we create? When am I done testing?
3. Communication: How do I as test manager communicate with the rest of the project and within my team. This includes giving and receiving feedback, basics of presentation techniques, maybe a bit of Myers-Briggs model
4. Explaining test to non-testers: Why are we not QA, why does it take so much time? Why do developers need to test as well. What is our goal as testers. What responsibilities do we NOT assume.
5. Managing a group of testers: Is it like being a project manager with testing skills? Important management principles: avoiding micro-management, coaching your staff, manager = problem remover, administrative tasks you need to do. Session-based test management
6. Effective bug management: templates, process, content, bug advocacy
7. Career plan for TM: what to read, what to do

So, fellow testers. What would be the three most important things YOU want to see in a test management class? Maybe the three things you wish they had told you before you agreed to your first job as a test manager.

And I have this great idea as well. What if we skilled context-driven testers together created some really good material for a test management class and made that material free to use for everyone? Like the BBST-class.  I really believe in sharing which is why my test design book in Swedish is available for free download at testzonen.se. Maybe it´s time to make the English version downloadable from thetesteye?

Cheers

-Tobbe

I am passionate about testing. This passion gives me energy,  the fire that makes me what to excel and get better at testing. It is this passion that I share with my co-workers, customers and my employees, hopefully growing their interest in testing and sharing my visions and goals. I use it to paint my views on a better tomorrow in the testing community, as I see it.

Would this passion then make us more biased in our testing than someone dispassionate? I think not. Everyone is biased from time to time.

Passion is synonym with ardent, blazing, burning, dithyrambic, fervent, fervid, fiery, flaming, glowing, heated, hot-blooded, hotheaded, impassioned, perfervid, red-hot, scorching and torrid. Dispassionate can be seen as not showing, and not affected by emotion, bias, or prejudice. Would it even be possible to compare the two in a testing duel?

My CEO has told me that he want competent and honest people in his company. He says, “Imagine having competent and dishonest people, that would be terrible.” I like those traits, but I would also like to add passionate as a trait.

As a customer or employer, would you want a tester who is passionate or a tester who is dispassionate about their beliefs in testing? Isn’t it so that we are just passionate over different things, which naturally is ok?

One of Cem Kaner’s many classic writings is “What is a Good Test Case?”
It is a very good article, well-spent time for anyone involved in software testing.
But when writing about test ideas, I started to realize that the list of properties for good test cases isn’t perfect, for me.
So it’s time for some criticism of this part of the professor’s article.

His list of attributes for good test cases (for the context “Tests Intended to Expose Defects”) goes something like this:

• Powerful – a test is more powerful if it is more likely to find bugs
• Yield significant results – the issues found are important (to stakeholders)
• Credible – the actions in the test are realistic (not corner cases “noone woud do”)
• Likely – the test reflects how customers actually will be using the product
• Easier to evaluate – ease of telling if the test passed or failed
• Useful for troubleshooting – so it is easy to find out what went wrong during the test
• Informative – regardless of pass/fail status, you get valuable information from the test
• Appropriately complex – if there are many bugs in the software, the test might fail too quickly
• Giving insightful information – the test might not render bugs, but other important information

Test cases can be seen as a broad spectra, from “classic” test cases with exact steps and expected results, to vague, one-liner test ideas (that also could be ongoing, unorthodox or unverifiable.)
So I have to disagree with “Easier to evaluate”, it’s a valuable property in many situations, but not in so many that it should make it on this list. But it’s back on the list if it is named something like “Hints for Evaluation”.

But now to the most interesting part of reviewing: what’s missing on the list?
There are many more things that could be important, a quick search gave:

• Ease of maintenance, Ease of creating variations of it (Shrini Kulkarni)
• Accurate, Economical, Reusable, Tracable, Self-cleaning (Dianne L. Runnels)

These are good things, but not needed on my list (and I’d prefer Fast to execute as name instead of Economical.)
But I have two other things I would like to add

• Easy to understand – to enable reviewing by different types of people (more likely to happen for one-liners)
• Enables serendipity – the test is rich in the sense it has possibilities of finding issues other than the ones the test case is aiming for

The serendipity can either be covered inside the test case, or by allowing variations, or by putting freedom and responsibility on all testers to deviate and look at more things, if deemed worthwile, while executing the test cases.

…and always remember Kaner’s advice that “Test cases can be “good” in a variety of ways. No test case will be good in all of them.”

Making estimates on a test assignment is hard. What you include and exclude varies between each person. As I see it, there are many things to consider that might make the whole test effort longer and make it more complex. So, instead of guessing or giving a random number consider this…

When you are asked how much effort you need to spend on the various aspects of testing for a new feature it is quite common you picture yourself that …

• it is the person him-/herself who does the estimation that will perform the assignment/testing
• each assignment is done by just one person instead of the whole or part of the test team
• the test team have good cooperation with product managers, developers, other teams, other testers that are involved in the assignment, no obstacles at all here that will steal time
• all hardware is available and is fully usable from start
• all hardware will remain in perfect shape, nothing ever get broken, nothing ever need to be repaired
• all software is available and is in perfect shape
• test instruments, used in testing, are fully functional with latest revisions of hardware and software
• each tester in the test team has tip top domain knowledge and no education about the assignment is needed, there is nothing out there that is complex or hard to understand
• each tester in the test team is executing tests on the correct test object, no confusion about revisions or if parts of the test object is correct
• the computers in the test lab are so fast and powerful that everything goes smooth at all times, even the network is fast at all times
• performing each test goes fast without any faults found, everything works perfectly
• there is no need for assisting developers pinpoint bugs, since none will be found
• there is no need to verify bugs found, since none will be found
• there is no need for performing any kind of regression, we did the right thing from the start
• all documentation that is interesting for the testers is in such good state that nothing is unclear
• nothing outside the plan for the test team ever happens during the whole project, you are always on track on all things
• the test planning is so good that it is always on time without any delays
• there is no need for any meetings or discussions inside the test team or outside, administration of the assignment is not needed at all
• there are no conflicts in the test team, never have been and never will be
• the test team is constant in size
• no one will ever be sick, have kids, have to sleep, go to the toilet or  have lunch breaks
• the test team is perfectly aligned in how they should work with testing
• the test team is perfectly aligned with how the team is managed, no confusion or downtime
• all test automation works perfectly at all times, no disinformation at all
• automated tests never break and always report pass
• automated tests does not need to be tested since the developers never make any mistakes
• the test team will not introduce any new tools or ways of working ever
• everyone in the test team estimate the same way and has the same idea on how it should be done
• personnel that is going to assist the test team also loves testing and will have no issues at all being removed from their previous position, it won’t affect their work at all
• personnel that is going to assist the test team will always bring joy and happiness to the current test team, there will be no conflicts ever
• utterly new personnel has full knowledge of corporate culture, where to get information, who does what and when, what is expected of them
• utterly new personel who stays for three weeks will be productive the whole time
• utterly new personnel never need anyone to mentor them or introduce them to assignments

… and you have not even gotten to the actual testing or test planning part.

I know… if you think like this will you ever get any assignment? Perhaps it is just better to give the amount of time that your stake holder want to hear? Still, I think that if you identify things that steal time from testing or make the assignment take a longer time, there is a big chance that management might be able to assist with removing these obstacles or see that you have thought this through.

Further reading related to test estimation:

http://blogs.msdn.com/testing123/archive/2007/04/11/demystifying-test-estimate.aspx

http://blogs.msdn.com/alanpa/archive/2007/03/13/test-guesstimation.aspx

http://www.developsense.com/blog/2009/11/why-is-testing-taking-so-long-part-1/

http://www.developsense.com/blog/2009/11/what-does-testing-take-so-long-part-2/

http://www.developsense.com/blog/2009/10/when-do-we-stop-testing-one-more-sure/

http://www.silverpath.com/resources/Silverpath-EstimatingTestEffortWhitepaper-080812.pdf

http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=8918&tth=DYN&tt=siteemail&iDyn=2

http://inderpsingh.blogspot.com/2010/03/how-to-estimate-testing-efforts-6.html

http://www.satisfice.com/blog/archives/124

and an infinite amount of good books on testing where most of them touch the test estimation subject.

“Testing can’t be complete” might be the only statement all testers would agree upon.
This means that we only will run a few of all possible tests, and this is in many fields called sampling.
There isn’t too much said about qualitative sampling in software testing, so let’s look at what Grounded Theory says about theoretical sampling, and the need for adjustments based on reality:

“To say that one samples theoretically means that sampling, rather than being predetermined before beginning the research, evolves during the process. It is based on concepts that emerged from analysis and that appear to have relevance to the evolving theory.” Strauss/Corbin p.202

“The general rule when building theory is to gather data until each category is saturated.” Strauss/Corbin p.212

“The analyst might be, and often is, surprised at what he or she finds out when in the field. Variations that the analyst expected to find might not be there. New ones might emerge from quite unexpected sources. The analyst also must be prepared for this. But this serendipity is what makes doing qualitative research and analysis so much fun. There always is that element of discovery.” Strauss/Corbin p.233

It has been said that “a good tester is often lucky“, and half of it is not true; a good tester has the ability to see and smell important things, there is no luck involved. The other half is semi-true; a good tester make preparations and deviations to have bigger chances of luck, and this makes them stumble on important information more often (serindipity). A skilled tester thinks about many aspects at the same time, and sees things she wasn’t explicitly looking for.

But let’s follow the sampling thought a bit longer.
If we agree that sampling and serendipity is needed, does it make more sense that we should use multiple approaches, in order to come close to all the important information?
Does it make no sense to script all tests in advance and aim for 100% Pass?
If you agree that quality is multi-dimensional , do you want to spend at least some thought and time for all orthogonal aspects?
All due respect to equivalance partitioning, but would you feel comfortable just ’cause some part of the functionality has pretty good coverage?

Maybe we can get most areas and aspects partly covered by samples, and spread the samples in a good way to have a nice shot at the serendipity we need to find most of the important information.
Maybe the spread doesn’t need to be theoretically perfect, by relying on tester skill and serendipity, we might go for the fastest, and richest, tests, so we can execute many of them.
As in Grounded Thory (and exploratory testing), we can change the sampling strategy as we learn more about the product.

Exercise: For the next 10 bugs you find, note if the issue was something you expected to find, or if you were looking for something else.
Or do it as a team effort, but only count really important issues, and make it a game:
Prediction (manufacturing) vs. Serendipity (qualitative analysis)