thoughts from the test eye

I believe that as a tester you do not value the same things as someone with a different role in the organisation. Each individual most certainly value different things, but I think there are some specific traits of the organisation that triggers a tester to stay or seek new employments.

As a test consultant you value different things. You move into different organisations knowing that you probably won’t stay long, still there are some areas that can be similar to what you value as a more permanent tester.

These are some of the things I value:

Test Approach: I value Exploratory test approach highly. If I enter an organisation with Script-based test approach I try to get it to move toward Exploratory.

Lovable product[s]: If I am to be an assest to the organisation I need to love the product or product family. At least have an interest in making the product something you take pride in.

Other testers: I think you work best as a team in testing. Therefore you co-workers become especially important. Do they have the same keen interest of testing as you? Do they also share the same passion?

Developers: I value the developers a lot. If you are working with good, open-minded, passionate developers it makes everything so much more fun.

Freedom: I value to have the freedom to do and say what I feel is right, even if it might hurt a bit for the organisation.

Trust: If I feel that my superiors and my co-workers trust in what I do and what I say, I will feel more confident and feel that my work brings value.

CEO: If the captain of the ship steers the vessel where I want to go, it makes a big difference for me as a tester. If the CEO think testers are important and that they add value, it is a key part for staying around.

Why did you seek an employment where you are today? When things are getting tough and it becomes hard to understand why you stay, what is keeping you?

What makes you stay?

It’s a truth that we can’t find all bugs, but is it really a truth that we can’t find all important bugs?
And it’s a cliche when used as answer to the (sincere) “why didn’t you find that bug?” question.
Testers are paid to find important information about what they are testing, and included are the big bugs, so you can fix them before the software is used in production.

Besides some good things about forthcoming test strategies, I hope you also can say: “if you have read the test plan and status reports it shouldn’t be a surprise bugs like this got away.” Well-informed risk taking is part of our business.
However, if there is a test that seems important, and would take a couple of minutes to perform, this argument is not valid. Testers (in a serious project) should try to perform all important tests that can be performed fast.

There are testing lessons to be learned if the answer is:
* we didn’t think about that scenario
* it wasn’t included in the requirements
* someone told us not to test that
* we had no idea that the customers where running that kind of configuration
* we found it, but didn’t report it well enough
* we didn’t put the bug there, and they hid it awfully well
* this type of issue can’t be found with automated tests

And there is the occasional invalid question, e.g. when the bug was reported in a good way, or when there was no testing time alotted for a late change.

Bottom line is: take the question seriously, and try to improve your testing to avoid this and other important issues (next time it will probably be another kind of issue…)

It has been said by many, but I heard it from Fiona Charles: “don’t ever fall in love with your model”, and this is a warning I want to elaborate.
A model is a powerful way to understand how the system works, and thereby also how it can fail.
But a model can also narrow your thinking, and stop you from testing outside the obvious patterns, in the ways your customers most certainly will.
So as a tester, you should use multiple (sometimes incoherent and contradictive) models, so you get a better coverage of the potential usage.
Having one model only is more dangerous than having none.

Here are some ideas that might help you use/create more models:

* requirements, and make sure to look for implicit as well as the explicit
* specifications are models, i.e. interpretations of what should be done; you’ll get different test ideas from functional, conceptual, technical and design specification
* the code is also a model, reading all of it is generally not feasible, but there might be some parts that are extra fruitful
* images; drawings, diagrams, mind maps, lists et.al. can show how things fit together, and if you don’t have them, you can create them (but don’t try to put everything in one place, rather use several with different angles, e.g. functional, architectural, technological, scenario-based…)
* test models – the way you think about, categorize and organize your tests, and the elements therein, is a model worth being aware of
* competitors products also model a solution to customers’ problems, and there might exist un-computerized solutions worth looking at
* the product in itself could be the best model; prototypes and early builds generate a lot of test ideas, especially because you can see how things can interact

You should be aware that each tester probably has his own implicit model of the system. This includes different aspects and is influenced by the functionality she has been testing, and which types of issues she has experienced, or heard about. It probably also includes a list of the most important quality characteristics, and it is a silent guide through the daily testing.
An implicit model is difficult to communicate, though.

With many models you can cover the product with different types of tests, and find more important things in the neverending flow of information.

First of all. I am honored to be invited as a guest writer at the testeye! I will start my contribution by asking for your assistance.

In the next year I have been asked to give a couple of classes for test managers that are fairly new in their role. I have been looking through the content description of a class I am supposed to take over and realize that I have a problem. So much of it seems to be picked from an ISTQB-class and just seem to be cementing the old ways of testing. I refuse to spend half a day talking about eight ways of measuring test which means measuring quality and is based on test case success ratio. And I am not going to dive into standards too much either.

So what am I going to put in a two day class for new test managers? These are some of the more important subjects I want to talk about.

1. Schools of testing: a test manager needs to understand that there are some fundamentally different ways that testing can be done. Since many of the students will be working as consultants they need to be prepared for the many different company approaches they may encounter.
2. Test Strategy: What is the testers’ mission. How do I plan my resources and my tasks. Time estimation. What do I need to know to do a good job. What kind of documentation should we create? When am I done testing?
3. Communication: How do I as test manager communicate with the rest of the project and within my team. This includes giving and receiving feedback, basics of presentation techniques, maybe a bit of Myers-Briggs model
4. Explaining test to non-testers: Why are we not QA, why does it take so much time? Why do developers need to test as well. What is our goal as testers. What responsibilities do we NOT assume.
5. Managing a group of testers: Is it like being a project manager with testing skills? Important management principles: avoiding micro-management, coaching your staff, manager = problem remover, administrative tasks you need to do. Session-based test management
6. Effective bug management: templates, process, content, bug advocacy
7. Career plan for TM: what to read, what to do

So, fellow testers. What would be the three most important things YOU want to see in a test management class? Maybe the three things you wish they had told you before you agreed to your first job as a test manager.

And I have this great idea as well. What if we skilled context-driven testers together created some really good material for a test management class and made that material free to use for everyone? Like the BBST-class.  I really believe in sharing which is why my test design book in Swedish is available for free download at testzonen.se. Maybe it´s time to make the English version downloadable from thetesteye?

Cheers

-Tobbe

I am passionate about testing. This passion gives me energy,  the fire that makes me what to excel and get better at testing. It is this passion that I share with my co-workers, customers and my employees, hopefully growing their interest in testing and sharing my visions and goals. I use it to paint my views on a better tomorrow in the testing community, as I see it.

Would this passion then make us more biased in our testing than someone dispassionate? I think not. Everyone is biased from time to time.

Passion is synonym with ardent, blazing, burning, dithyrambic, fervent, fervid, fiery, flaming, glowing, heated, hot-blooded, hotheaded, impassioned, perfervid, red-hot, scorching and torrid. Dispassionate can be seen as not showing, and not affected by emotion, bias, or prejudice. Would it even be possible to compare the two in a testing duel?

My CEO has told me that he want competent and honest people in his company. He says, “Imagine having competent and dishonest people, that would be terrible.” I like those traits, but I would also like to add passionate as a trait.

As a customer or employer, would you want a tester who is passionate or a tester who is dispassionate about their beliefs in testing? Isn’t it so that we are just passionate over different things, which naturally is ok?

One of Cem Kaner’s many classic writings is “What is a Good Test Case?”
It is a very good article, well-spent time for anyone involved in software testing.
But when writing about test ideas, I started to realize that the list of properties for good test cases isn’t perfect, for me.
So it’s time for some criticism of this part of the professor’s article.

His list of attributes for good test cases (for the context “Tests Intended to Expose Defects”) goes something like this:

• Powerful – a test is more powerful if it is more likely to find bugs
• Yield significant results – the issues found are important (to stakeholders)
• Credible – the actions in the test are realistic (not corner cases “noone woud do”)
• Likely – the test reflects how customers actually will be using the product
• Easier to evaluate – ease of telling if the test passed or failed
• Useful for troubleshooting – so it is easy to find out what went wrong during the test
• Informative – regardless of pass/fail status, you get valuable information from the test
• Appropriately complex – if there are many bugs in the software, the test might fail too quickly
• Giving insightful information – the test might not render bugs, but other important information

Test cases can be seen as a broad spectra, from “classic” test cases with exact steps and expected results, to vague, one-liner test ideas (that also could be ongoing, unorthodox or unverifiable.)
So I have to disagree with “Easier to evaluate”, it’s a valuable property in many situations, but not in so many that it should make it on this list. But it’s back on the list if it is named something like “Hints for Evaluation”.

But now to the most interesting part of reviewing: what’s missing on the list?
There are many more things that could be important, a quick search gave:

• Ease of maintenance, Ease of creating variations of it (Shrini Kulkarni)
• Accurate, Economical, Reusable, Tracable, Self-cleaning (Dianne L. Runnels)

These are good things, but not needed on my list (and I’d prefer Fast to execute as name instead of Economical.)
But I have two other things I would like to add

• Easy to understand – to enable reviewing by different types of people (more likely to happen for one-liners)
• Enables serendipity – the test is rich in the sense it has possibilities of finding issues other than the ones the test case is aiming for

The serendipity can either be covered inside the test case, or by allowing variations, or by putting freedom and responsibility on all testers to deviate and look at more things, if deemed worthwile, while executing the test cases.

…and always remember Kaner’s advice that “Test cases can be “good” in a variety of ways. No test case will be good in all of them.”

Making estimates on a test assignment is hard. What you include and exclude varies between each person. As I see it, there are many things to consider that might make the whole test effort longer and make it more complex. So, instead of guessing or giving a random number consider this…

When you are asked how much effort you need to spend on the various aspects of testing for a new feature it is quite common you picture yourself that …

• it is the person him-/herself who does the estimation that will perform the assignment/testing
• each assignment is done by just one person instead of the whole or part of the test team
• the test team have good cooperation with product managers, developers, other teams, other testers that are involved in the assignment, no obstacles at all here that will steal time
• all hardware is available and is fully usable from start
• all hardware will remain in perfect shape, nothing ever get broken, nothing ever need to be repaired
• all software is available and is in perfect shape
• test instruments, used in testing, are fully functional with latest revisions of hardware and software
• each tester in the test team has tip top domain knowledge and no education about the assignment is needed, there is nothing out there that is complex or hard to understand
• each tester in the test team is executing tests on the correct test object, no confusion about revisions or if parts of the test object is correct
• the computers in the test lab are so fast and powerful that everything goes smooth at all times, even the network is fast at all times
• performing each test goes fast without any faults found, everything works perfectly
• there is no need for assisting developers pinpoint bugs, since none will be found
• there is no need to verify bugs found, since none will be found
• there is no need for performing any kind of regression, we did the right thing from the start
• all documentation that is interesting for the testers is in such good state that nothing is unclear
• nothing outside the plan for the test team ever happens during the whole project, you are always on track on all things
• the test planning is so good that it is always on time without any delays
• there is no need for any meetings or discussions inside the test team or outside, administration of the assignment is not needed at all
• there are no conflicts in the test team, never have been and never will be
• the test team is constant in size
• no one will ever be sick, have kids, have to sleep, go to the toilet or  have lunch breaks
• the test team is perfectly aligned in how they should work with testing
• the test team is perfectly aligned with how the team is managed, no confusion or downtime
• all test automation works perfectly at all times, no disinformation at all
• automated tests never break and always report pass
• automated tests does not need to be tested since the developers never make any mistakes
• the test team will not introduce any new tools or ways of working ever
• everyone in the test team estimate the same way and has the same idea on how it should be done
• personnel that is going to assist the test team also loves testing and will have no issues at all being removed from their previous position, it won’t affect their work at all
• personnel that is going to assist the test team will always bring joy and happiness to the current test team, there will be no conflicts ever
• utterly new personnel has full knowledge of corporate culture, where to get information, who does what and when, what is expected of them
• utterly new personel who stays for three weeks will be productive the whole time
• utterly new personnel never need anyone to mentor them or introduce them to assignments

… and you have not even gotten to the actual testing or test planning part.

I know… if you think like this will you ever get any assignment? Perhaps it is just better to give the amount of time that your stake holder want to hear? Still, I think that if you identify things that steal time from testing or make the assignment take a longer time, there is a big chance that management might be able to assist with removing these obstacles or see that you have thought this through.

Further reading related to test estimation:

http://blogs.msdn.com/testing123/archive/2007/04/11/demystifying-test-estimate.aspx

http://blogs.msdn.com/alanpa/archive/2007/03/13/test-guesstimation.aspx

http://www.developsense.com/blog/2009/11/why-is-testing-taking-so-long-part-1/

http://www.developsense.com/blog/2009/11/what-does-testing-take-so-long-part-2/

http://www.developsense.com/blog/2009/10/when-do-we-stop-testing-one-more-sure/

http://www.silverpath.com/resources/Silverpath-EstimatingTestEffortWhitepaper-080812.pdf

http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=8918&tth=DYN&tt=siteemail&iDyn=2

http://inderpsingh.blogspot.com/2010/03/how-to-estimate-testing-efforts-6.html

http://www.satisfice.com/blog/archives/124

and an infinite amount of good books on testing where most of them touch the test estimation subject.

“Testing can’t be complete” might be the only statement all testers would agree upon.
This means that we only will run a few of all possible tests, and this is in many fields called sampling.
There isn’t too much said about qualitative sampling in software testing, so let’s look at what Grounded Theory says about theoretical sampling, and the need for adjustments based on reality:

“To say that one samples theoretically means that sampling, rather than being predetermined before beginning the research, evolves during the process. It is based on concepts that emerged from analysis and that appear to have relevance to the evolving theory.” Strauss/Corbin p.202

“The general rule when building theory is to gather data until each category is saturated.” Strauss/Corbin p.212

“The analyst might be, and often is, surprised at what he or she finds out when in the field. Variations that the analyst expected to find might not be there. New ones might emerge from quite unexpected sources. The analyst also must be prepared for this. But this serendipity is what makes doing qualitative research and analysis so much fun. There always is that element of discovery.” Strauss/Corbin p.233

It has been said that “a good tester is often lucky“, and half of it is not true; a good tester has the ability to see and smell important things, there is no luck involved. The other half is semi-true; a good tester make preparations and deviations to have bigger chances of luck, and this makes them stumble on important information more often (serindipity). A skilled tester thinks about many aspects at the same time, and sees things she wasn’t explicitly looking for.

But let’s follow the sampling thought a bit longer.
If we agree that sampling and serendipity is needed, does it make more sense that we should use multiple approaches, in order to come close to all the important information?
Does it make no sense to script all tests in advance and aim for 100% Pass?
If you agree that quality is multi-dimensional , do you want to spend at least some thought and time for all orthogonal aspects?
All due respect to equivalance partitioning, but would you feel comfortable just ’cause some part of the functionality has pretty good coverage?

Maybe we can get most areas and aspects partly covered by samples, and spread the samples in a good way to have a nice shot at the serendipity we need to find most of the important information.
Maybe the spread doesn’t need to be theoretically perfect, by relying on tester skill and serendipity, we might go for the fastest, and richest, tests, so we can execute many of them.
As in Grounded Thory (and exploratory testing), we can change the sampling strategy as we learn more about the product.

Exercise: For the next 10 bugs you find, note if the issue was something you expected to find, or if you were looking for something else.
Or do it as a team effort, but only count really important issues, and make it a game:
Prediction (manufacturing) vs. Serendipity (qualitative analysis)

I have become allergic to models that are brought from other industries, and put on software testing as a best practice, or something really good.
Software testing is unique, and you might violate important aspects when applying a template that doesn’t match.
It is a big difference between producing 100,000 cars a year, and one piece of new software.

As I believe that software testing is one of the most creative professions, maybe there are other places to look for really good management methodologies?
The Beatles wrote great pop songs, and a lot of them, where they using Kanban?
Did Just-in-Time play a role for the productivity of Thomas Alva Edison?
Was Leonardo da Vinci an early adopter of Kaizen?
Were the New Wave in French Film under Lean management?
Could the String Theory emerge without Six Sigma?

You might argue that these examples are very different from software testing, and that is true.
Therefore we should manage and improve our work in ways that are suitable to what we do, and who we are.
There are no frameworks for passion.

Let’s have a look at an example:
A key element in Lean is “reduce waste”, which seems like a natural and good thing to do.
But this is very problematic in testing, because you don’t know (in advance) what is waste. And what was waste in one situation might be very important in another. It might be reasonable to identify waste in activities that are predictable, but it’s not recommendable in situations with a lot of serendipity.
Of course it can be a good match as well, but I’d prefer saying “I read some about car manufacturing, and realized that parts of our test report aren’t useful to anyone, so we will skip those parts” to “let’s adopt Lean Management to our testing process!”

It’s a good idea to be inspired from other fields, but please don’t take a whole package, and please think carefully if there are things about testing that are necessary to consider, e.g. doctors can not make mistakes, testers should make mistakes.

I’m thinking movie production might be the closest to “normal” software testing projects: it involves a lot of people, a lot of different types of creativity, is complex, and spans over quite some time. What management models are helpful in the film industry?
Does it help if movie requirements involve things like “There must be exactly 3,5 minutes of action scenes including tricycles”?

Typecasting is the process by which a film, TV, or stage actor is strongly identified with a specific character, one or more particular roles, or characters with the same traits or ethnic grouping. For many actors this has been a nightmare, even if they have earned a fortune on it.

I believe that some of us testers are being typecast; and even though we might be good at what we do in our role, there is a problem for many to change their role once they have been typecast.

Sean Connery was once asked how to avoid being typecast, he said, “First you have to be good enough that they ask you to play it again and again”. And this could very well apply to typecast in testing; and notice that Sean’s explanation “good enough” means that you play the role good enough even if your role is to be “unskilled”.

Here are some examples on when you have become typecast; and why you should try to break out.

1. The novice tester
   We are all novice testers in the beginning, but for some testers, this label won’t go away. When work packages are dealt out in the team and you feel that you are always treated like the novice tester, it is time for you to step up and move away from this.
2. The technical expert
   We all have different backgrounds that we bring into testing. For some testers this has been a burden that they can’t get away from. Let’s say that you are a database expert or application server expert, then it is very likely that you will work with this in testing as well; at least in the beginning. But you know when you have been typecast when you always get to do the tests against the database even though you would like to try out other areas; you know that you have been typecast when you always are assigned to do the platform tests on different application servers even though you hate it.
3. The specific-persona tester
   Again, testers coming from different backgrounds might also affect you by always getting to the play the role of a certain persona when performing usage-centric testing (scenario testing, soap-opera testing, persona-based testing, etc). You know when you have become typecast when you with your clerk background always have to put on the clerk-persona mask every time there is a need for testing from a user perspective. Or if you come from sales and have been on the field for several years, there is a big chance that you always will be the sales-persona speaking for all sales persons in the field. Besides being boring for you, this is dangerous for at least two other reasons: the others make you responsible for all users of your category which enable for them to blame on you: “X said that this was what the users would expect”; and the others don’t need to think about these users and the problems that they might face leaving you with the task to come up with all problems yourself.
4. The unskilled tester
   Similar to the Novice tester, you will end up doing all the easy and/or boring stuff that no one else need to do. This might happen because you once were unskilled, but even though you have evolved over the years you are still treated as the unskilled tester. You know when you have been typecast as the unskilled tester when your tasks always are the uncomplicated ones; or when you always have to do regression testing that no one else want to do. Also, as tempting as it might be sometimes, growing your strategic incompetence might as well make you end up being typecast as the unskilled tester – for life.
5. The bug verifier
   Even though it is good to learn about the product by verifying other peoples bugs, you might become so good at this that you are being typecast as the bug verifier. On every new build, you are assigned to verify bugs that have been fixed in the build, while the rest of the team start to test the new (and interesting) functionality. Being typecast as the bug verifier will inhibit you from a lot of creative thinking which is so important in testing; and this might make you blasé and start becoming inattentional blind to important things in the periphery.
6. The super tester
   Being a super tester is nice, but sometimes this might be a burden upon your shoulders. You are always expected to find all the hard things that no one else seems to find. Not only can this be too much of a burden for you, but it also enables for other testers in the team to say “Don’t bother with that, it is not our responsibility. Let mr Y have a look at it…”. Being typecast as the super tester might also cause a lot of negative stress e.g. if your social life is chaotic. All sapient testers are human beings, with all that comes with that.

Typecasting in testing is an easy way to stop your team from growing and developing. It is a way of letting people win the “Not my job”-award by not taking responsibility (or not been given responsibility). It is a way to foster specialists that might want to be generalists.

In order for you to get away from being typecast, you need to do some change.
You should try as many ways as possible, here are some suggestions:

• specialize in new areas (where you will become the expert in the team);
• learn more about testing – read books and take courses in testing;
• talk to the team leader about you wanting to learn more about the areas where you are currently kept away from – remember that all experts have once started as novices;
• partner up with your colleagues and work in pairs;
• make sure that you are involved in all parts of the testing;
• change employer;
• etc

But I guess that the most important thing is to realize if, and when, you are being typecast; and not being satisfied with that. Then you at least have a chance to break out.

Also see The Generalist vs. Specialist Paradox