thoughts from the test eye

I have become allergic to models that are brought from other industries, and put on software testing as a best practice, or something really good.
Software testing is unique, and you might violate important aspects when applying a template that doesn’t match.
It is a big difference between producing 100,000 cars a year, and one piece of new software.

As I believe that software testing is one of the most creative professions, maybe there are other places to look for really good management methodologies?
The Beatles wrote great pop songs, and a lot of them, where they using Kanban?
Did Just-in-Time play a role for the productivity of Thomas Alva Edison?
Was Leonardo da Vinci an early adopter of Kaizen?
Were the New Wave in French Film under Lean management?
Could the String Theory emerge without Six Sigma?

You might argue that these examples are very different from software testing, and that is true.
Therefore we should manage and improve our work in ways that are suitable to what we do, and who we are.
There are no frameworks for passion.

Let’s have a look at an example:
A key element in Lean is “reduce waste”, which seems like a natural and good thing to do.
But this is very problematic in testing, because you don’t know (in advance) what is waste. And what was waste in one situation might be very important in another. It might be reasonable to identify waste in activities that are predictable, but it’s not recommendable in situations with a lot of serendipity.
Of course it can be a good match as well, but I’d prefer saying “I read some about car manufacturing, and realized that parts of our test report aren’t useful to anyone, so we will skip those parts” to “let’s adopt Lean Management to our testing process!”

It’s a good idea to be inspired from other fields, but please don’t take a whole package, and please think carefully if there are things about testing that are necessary to consider, e.g. doctors can not make mistakes, testers should make mistakes.

I’m thinking movie production might be the closest to “normal” software testing projects: it involves a lot of people, a lot of different types of creativity, is complex, and spans over quite some time. What management models are helpful in the film industry?
Does it help if movie requirements involve things like “There must be exactly 3,5 minutes of action scenes including tricycles”?

Typecasting is the process by which a film, TV, or stage actor is strongly identified with a specific character, one or more particular roles, or characters with the same traits or ethnic grouping. For many actors this has been a nightmare, even if they have earned a fortune on it.

I believe that some of us testers are being typecast; and even though we might be good at what we do in our role, there is a problem for many to change their role once they have been typecast.

Sean Connery was once asked how to avoid being typecast, he said, “First you have to be good enough that they ask you to play it again and again”. And this could very well apply to typecast in testing; and notice that Sean’s explanation “good enough” means that you play the role good enough even if your role is to be “unskilled”.

Here are some examples on when you have become typecast; and why you should try to break out.

1. The novice tester
   We are all novice testers in the beginning, but for some testers, this label won’t go away. When work packages are dealt out in the team and you feel that you are always treated like the novice tester, it is time for you to step up and move away from this.
2. The technical expert
   We all have different backgrounds that we bring into testing. For some testers this has been a burden that they can’t get away from. Let’s say that you are a database expert or application server expert, then it is very likely that you will work with this in testing as well; at least in the beginning. But you know when you have been typecast when you always get to do the tests against the database even though you would like to try out other areas; you know that you have been typecast when you always are assigned to do the platform tests on different application servers even though you hate it.
3. The specific-persona tester
   Again, testers coming from different backgrounds might also affect you by always getting to the play the role of a certain persona when performing usage-centric testing (scenario testing, soap-opera testing, persona-based testing, etc). You know when you have become typecast when you with your clerk background always have to put on the clerk-persona mask every time there is a need for testing from a user perspective. Or if you come from sales and have been on the field for several years, there is a big chance that you always will be the sales-persona speaking for all sales persons in the field. Besides being boring for you, this is dangerous for at least two other reasons: the others make you responsible for all users of your category which enable for them to blame on you: “X said that this was what the users would expect”; and the others don’t need to think about these users and the problems that they might face leaving you with the task to come up with all problems yourself.
4. The unskilled tester
   Similar to the Novice tester, you will end up doing all the easy and/or boring stuff that no one else need to do. This might happen because you once were unskilled, but even though you have evolved over the years you are still treated as the unskilled tester. You know when you have been typecast as the unskilled tester when your tasks always are the uncomplicated ones; or when you always have to do regression testing that no one else want to do. Also, as tempting as it might be sometimes, growing your strategic incompetence might as well make you end up being typecast as the unskilled tester – for life.
5. The bug verifier
   Even though it is good to learn about the product by verifying other peoples bugs, you might become so good at this that you are being typecast as the bug verifier. On every new build, you are assigned to verify bugs that have been fixed in the build, while the rest of the team start to test the new (and interesting) functionality. Being typecast as the bug verifier will inhibit you from a lot of creative thinking which is so important in testing; and this might make you blasé and start becoming inattentional blind to important things in the periphery.
6. The super tester
   Being a super tester is nice, but sometimes this might be a burden upon your shoulders. You are always expected to find all the hard things that no one else seems to find. Not only can this be too much of a burden for you, but it also enables for other testers in the team to say “Don’t bother with that, it is not our responsibility. Let mr Y have a look at it…”. Being typecast as the super tester might also cause a lot of negative stress e.g. if your social life is chaotic. All sapient testers are human beings, with all that comes with that.

Typecasting in testing is an easy way to stop your team from growing and developing. It is a way of letting people win the “Not my job”-award by not taking responsibility (or not been given responsibility). It is a way to foster specialists that might want to be generalists.

In order for you to get away from being typecast, you need to do some change.
You should try as many ways as possible, here are some suggestions:

• specialize in new areas (where you will become the expert in the team);
• learn more about testing – read books and take courses in testing;
• talk to the team leader about you wanting to learn more about the areas where you are currently kept away from – remember that all experts have once started as novices;
• partner up with your colleagues and work in pairs;
• make sure that you are involved in all parts of the testing;
• change employer;
• etc

But I guess that the most important thing is to realize if, and when, you are being typecast; and not being satisfied with that. Then you at least have a chance to break out.

Also see The Generalist vs. Specialist Paradox

Testers are often in a unique position because we know a lot about the system as a whole, but also a lot about the details of the operating software.
There are interesting dynamics between the small and the large, and with a human mind in between, a lot of important information will emerge.

“The distinction between micro and macro is an artificial one.” [1]

The whole system consists of the many details; the perceptions of the details are based on the expectations of the whole system.
“Everything is connected” [2], “The devil is in the details” [3]

Enough fluff; how can a tester take advantage of this artificial dynamics?

1) Become an expert of your macro
Find out (for real!) what the users want to accomplish with your software. Learn the even bigger box; what values (and in which ways) are expected from your customers’ customers?

2) Become an expert of your micros
You can take the technical path; learn the details of how a part of your system works and the details about the surrounding it interacts with, and (create) all the tools needed for your full advantage.
Or take the quality path; delve into details of a quality characteristic, learn about any types of deviations, learn which sub-aspects are really important for your software, and have this in the back of your head whatever you are testing.

Or if possible, do all of the above, and provide value; super-fast and all-the-time.

This is a follow up from previous articles on Growing test teams based on the ideas from Peopleware by Tom DeMarco and Timothy Lister.

Uncertain team composition

If you are newly assigned to be a team leader there is a big chance that you also have a team, but that is not always the case. Just as you want to start organizing the team, steer toward a set of goals, begin promising your stakeholders what you will be able to accomplish… you begin to wonder… why am I sitting alone on this team meeting? Where are they?

The test team will have a harder time growing when …

• you do not know who is in the team
• you do not know how much time in the resource plan each member is allocated to the team
• you do not know how much time in reality each member is allocated to the team
• your team members have other assignments that they spend time on that are not communicated
• your test manager does not see you as a team, but instead a resource pool where all members are to be plucked out at any time
• your team members does not see themselves as part of your team and continue to work on their own agendas
• you have hidden members that should be part of the team, but are safely tucked away in a hidden project

When you are uncertain about your team composition it will block you from going forward. You will be handicaped as a team leader and will most certainly fail, or burn out trying.

There are many names for software testing strategies/activities/ approaches/processes; they can be risk-based, coverage-focused, exploratory, requirements-based, Super-TPI, TMM 5 et.al.
The names generally come from how the testing is performed or initiated, so I thought we should look at it from another angle, from the end of testing, from the results that we might know about a year later.
I like the content of Good Enough Testing, but the words are easily misunderstood; we don’t have definitions for Crappy or Decent Testing, and since that often isn’t enough it might be useful with Saturated Testing. (This is elaborated elsewhere with a different name??)

The term is borrowed from Grounded Theory (social science qualitative analysis), and means that further research in an area doesn’t give any important, new information, and therefore isn’t worth the effort.
The same can happen in ambitious software testing (if the test time isn’t radically shortened…): when important bugs no longer are found, and all important test areas have been covered, and no code changes are made, there is no need to continue testing.

I see three hypothetical levels of Testing Saturation:
1) more testing will not find information that would change the release decision
2) more testing will not find issues that later would be patched
3) more testing will not find bugs that we would like to have fixed

If I you use a mix of these as your goal, I have three warnings:
* Have you used all relevant information?
* Have you tried all relevant test approaches?
* Have enough people with different views been involved in (at least) generating test ideas?

To get a result you are satisfied with, you need to think about things like this early, you need a lot of skill and hard work, and a fair share of luck.

Well, to many people this is nothing new. But still, there are a lot of testers, and indeed test leads, that still think that Exploratory Testing is a technique that can be used in testing. To some extent, it has to do with that both Cem Kaner and James Bach have used this term amongst other techniques (e.g., in the BBST course material). But they have changed and updated presentations as much as possible over the last period of time.

According to Cem Kaner nowadays, the definition of exploratory testing is “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

And this is important. You can come a long way on reaching the style of Exploratory Testing just by treating testers as intelligent people; which is one of the most important factors in the definition above. In contrast to Exploratory Testing you have Scripted Testing that, in my opinion, treats testers as dumb people or even dumb machines. I think that this approach is devastating for our profession (even though I can somehow see the need for Scripted Testing in some places).

A technique is a recipe for solving a problem, whereas a style (or approach) is a way of thinking around a theme that stretches far beyond solving a particular problem.
So when we talk about selling in Exploratory Testing to managers or project stakeholders it is not a technique that we are selling; it is rather an acceptance of a mindset where testers are treated as professional and intellectual human beings that are able to perform Sapient Testing, and particularly in an Exploratory way. It is not about stakeholders investing in a technique, it is about them showing that they have as much trust in testers as they have in other intelligent co-workers of the project.

Exploratory Testing is not as widely used as it could be, because management doesn’t want it.
Stated reasons for this could be unaccountable, unstructured, sloppy, non-scientific etc, reasons that can be refuted by communication.
But I think the real reason is something Exploratory Testing can’t have: a controlled process.

Management/Companies want to have a plan with dates and costs; they want a test manager to be able to say how many percent of the testing is completed; they incorrectly think that software development is a lot like manufacturing.
Exploratory Testing can’t have this, because the testing will change as new information is uncovered.
Testing might be quick, might take a long, long time, might not be close to complete, might look outside the requirements, and discover so important information that the whole project needs to be re-thinked.
These are not good things for some managers, they want control and precision.
They want to see everything go smoothly and at the promised date we deliver what we said we would deliver.
The focus is on the control, not the result.
That is why managers can prefer to outsource testing to a company that runs the same script over and over until they all pass, than to a company (or ideally, in-house testers) that will change their testing strategies along the way, and leave the project with a bunch of fixed and unfixed bugs.

Exploratory testing is about learning, discovering new things and changing our mind.
And for software testing, which can’t be complete, this is a very good thing.
It enables better products, in a way that can be managed, but not controlled.

This is why it is so hard to sell Exploratory Testing:
First you must sell trust, which is priceless.

It is good to strive for better requirements by critical analysis (and looking for what’s missing), but there is a danger in complaining about untestable requirements.
If those vague requirements are changed (made too specific) or removed, the words in the requirements document have less meaning, and less chance of guiding towards great software.

And there are no such things as untestable requirements. There are requirements that can’t be verified, that can’t get a true/false stamp, but you can definitely test the software and look for things that don’t match the essence of the vagueness.

An example: “the feature should be easy to operate” is difficult to prove right or wrong, but very easy to evaluate subjectively after doing some manual testing.
If the requirement is changed to “minimum no. of mouse-clicks to perform common operations”, you might catch some issues, but some other, more important things, might be lost in translation.
And if the requirements are split into many, many smaller pieces, you might lose less information, but end up with a too complex document that is very expensive to create and maintain.
It’s not a bad thing to be specific, but that’s not feasible for everything.

There’s an underlying assumption I should tell you about:
I do not think requirements should be contractual, they should rather be aiding – they should help the development team produce good software.
Since requirements neither can be complete nor perfect, we should rather take advantage of oppurtunities that arise, and create something that can solve problems. If the essence of the unspoken requirements are captured, it might not matter that a few specifics aren’t met.

Testers should keep in mind that there’s a greater whole we’re aiming for, and do our best with what we have, so be it unverifiable requirements.

In order to determine where you are heading with your test department it is good to understand where you are currently standing as a group and as individuals in the group.

Understand which way of working with quality that you tend to lean the most against. Use Brett Petticord’s Four Schools of Testing [1] as a start point for discussion. Did this rattle your thoughts on testing in any direction or in many directions? Do you have conflicting ideological beliefs about testing within the test group? Has this set the test group in motion in any direction?

What view do you have on testers in your organizations? Can anyone be a tester? Can anyone in the organization assist testing? Are you expected to create test cases that anyone can use when performing a test? Perhaps it is so that many in the organization think that the intelligence is in the test script, not the tester executing the script? Cem Kaner’s Ongoing Revolution of Software Testing [2] might shed some light on the subject? If you are viewed upon as a group of professional testers that is great, if not… what do you intend to do about it?

What view does the test manager have? Is he/she a former tester with experience from your business? Is he/she inexperienced with testing and is more of administrator? Does he/she make decisions about the test process and test strategies? Perhaps it is time you get him/her involved in what you do and where you think the test department should go?

If you are going in a specific direction, what is pushing or pulling you there? In some cases the company/organization is moving toward a specific goal or a new way of working, does this mean that your test department must change or move toward the same new goals? What platform do your company [3] lean towards? Is it the fundamentals from Scientific Management [4] or is from Management style according to Drucker [5].

If the company is trying to go towards Lean/Agile, what obstacles do you see if the organization is based on Taylor’s ideas? Do organizational structures, internal tools (resource allocation, time reporting, etc), project models, general thoughts and expectations stand between those goals? Do you think it will be a bumpy ride [6] or not [7], perhaps something else [8]?

What is the focus on education for the testers? Is there focus on ISTQB or Exploratory Testing, perhaps a bit of both? Does the basis for education align with where you are going or want to go?

Now… where do you want to go next?

References:

[1] http://www.testingeducation.org/conference/wtst_pettichord_FSofST2.ppt

[2] http://www.kaner.com/pdfs/TheOngoingRevolution.pdf

[3] http://www.usnews.com/usnews/biztech/articles/030224/24manage.htm

[4] http://www.ibiblio.org/eldritch/fwt/taylor.html

[5] http://en.wikipedia.org/wiki/Peter_Drucker

[6] http://www.claretyconsulting.com/it/comments/agile-scrum-fails-to-get-to-grips-with-human-psychology/2006-08-17/

[7] http://martinfowler.com/articles/newMethodology.html

[8] http://www.modernanalyst.com/Resources/Articles/tabid/115/articleType/ArticleView/articleId/1153/The-Next-Wave-Valuable-Products-First-Process-Second.aspx

Today we celebrate our 100th post on this blog!

It has been an interesting journey for us so far; and we realize that we have only begun this ride, a ride with no destination but to enrich ourselves with wisdom and knowledge through discussions and by sharing thoughts. And you, our readers, are a very important part of this process.

Our decision to write posts in English has been a good strategy as we see it; not only do we reach people all over the world, but we can also discuss matters with peers that, in this community, don’t have any country borders. But you have to bear with us sometimes if you think that our English isn’t perfect, we are working on it… We do hope that the message gets through though.

The thing that we are three persons sharing a passion for testing and a blog to express it, has been a really good thing as we see it. Not only is it more fun, but it is also some assurance that the blog will keep up a good tempo; since there is always someone of us that has inspiration and time to write. And it will also be a guarantee that the topics that we choose to write about are evolving and often diversified. Over the years we have noticed that we agree on many things, but disagree or have different perspective on as many.

We are very happy to see that we now have a really large group of readers from all over the world, and we have had a huge increase of new readers over the last two years. We hope that you will keep finding us interesting and continue reading in the future; and we invite you to contribute even more with your comments on our thoughts.

Do you have any thoughts on where you think we should go from here on?

Best regards,

Henrik, Martin & Rikard
the test eye