thoughts from the test eye

I do not see myself as a member of any of the Schools of Testing, and I have ethical problems with labelling other people than yourself.
However, I see the schools as a fruitful tool for enhancing your understanding of views on testing.
So please join me in the following thought experiment.

The following is not a perfect match of my personal opinions, it is fiction. I try to be a bit funny and serious at the same time, and I know this is easily misunderstood.

Suppose I run a consultency firm specialized in testing.
We take any kind of testing job (we can also do checking only if requested) and we are very successful; we provide a lot of valuable information to our clients.
The key to our success comes from this recipe:

1. A lot of exploratory testing
ET gives better results, and by explicitly giving testers responsibility and freedom for their activities, they stay motivated and get better over time.
It happens that clients question this approach, but usually it ís sufficient to point at the many Internet resources, and say that they have to be modern, this is the greatest super-good practice right now.

2. Training scheme
We hire people that are fast learners, curious and ambitious. All must take AST courses Foundation, Bug Advocacy, Test Design (the bug reporting is essential, we impress our clients early on, and is the reason we can offer a Money-Back-Guarantee.)
Then everyone must go Rapid Software Testing with Bach or Bolton, which inspires and give breadth to the thinking.
Finally they take Foundation and Advanced ISTQB certification (this is last to avoid problems at previous classes.)
We are no fans of ISTQB, but it is good to know what others know, and we eliminate the risk of losing a deal on a (ridicilous) client requirement.
We also do continuous training on collaboration and feedback, after all, people are the most important part of any context.

3. SBTM
We manage the testing efforts by three sessions a day, this squeezes out maximum from the testers without wearing them out.
We log all the proposed statistics so we can show numbers and progress.
We always add 24% to originally planned sessions, and make sure this cover unexpected things.
Planning and other processes are run context-wise, we use whatever the client is using.

4. Standards
For all jobs, we walk the client through thetesteye’s Quality Characteristics, and a secret company standard for Multiple Information Sources.
The client decides what’s most important in dialogue with us. (We of course also do at least one test per requirement.)
Testers use HICCUPPS(F), CRUCSPIC STMP, CIDTESTD and SFDPOT in their daily work, we know it’s not perfect, but easy to remember.

5. Reasonable Resource Allocation
In a typical project we want to have 1 tester per 3 developers. We tell clients this is our standard where we know we can do a good job. We can do higher or lower ratio, but clearly communicate which expectations this should give.
This makes it easy for clients to measure cost; it has a direct relationship with no. of developers, development time, and start of test effort (the sooner the better, but clients decide.)
Since we have a common background and practices, it is easy to replace our factory workers if needed. But we always keep 1 person from the old staffing, so crucial knowledge can be transferred.
Scaling up or down is not a problem, we have a pool of skilled resources, that grows over time as we hire more talents.

I feel we have a highly manageable process, that repeatedly bring good results. Sure, all projects are unique, but this scheme has proven to be very good, so far.

So what do you think dear readers, would this be a good approach?
Is it Factory, Context-Driven, or something else?

The delegates of the second Swedish Workshop on Exploratory Testing (Test Planning and Status Reporting for Exploratory Testing) were:
Henrik Andersson, Azin Bergman, Robert Bergqvist, Sigge Birgisson, Rikard Edgren, Henrik Emilsson, Ola Hyltén, Martin Jansson, Johan Jonasson, Saam Koroorian, Simon Morley, Torbjörn Ryber, Fredrik Scheja, Christin Wiedemann, Steve Öberg

Discussions on peer conferences can’t be summarized, but you can read the abstracts, Ryber’s notes, and here comes some quote highlights:

- How many test cases do you want? 800? Then let’s say we have 800. (Jonasson)
- Think “there are problems”, not “there might be problems” (Bergqvist)
- To motivate: lead by example, avoid de-motivating (Hyltén, Andersson)
- They want to see bug curves, but don’t know it going up or down is good or bad. (Jonasson)
- Rather ask “What are you afraid of?” (Jansson)
- Qualitative information is sensitive to filtering (Emilsson)
- Dialogue is important! (Morley)
- Exploratory test planning, adapt to reality. That’s how we all work. (Scheja)
- A test expert that doesn’t know anything (favorite slip of the tongue from Jansson)
- Has this story confirmed that anyone can test? (Bergman)
- Incorrect expectations: if we test, there won’t be any production problem (Ryber)
- Categorizations are good for learning, but throw them away to meet reality (Edgren)
- I always use low-tech testing dashboards, except in this case (Ryber)
- A tester is also test leader, test designer, test planner (Scheja)
- It is small raisins in a very large cake (Jonasson)
- I want to test until the world is on fire (Wiedemann)
- Care about what is most important (Emilsson)
- I invent many wheels every day (Scheja)
- I want more emphasis on which information we have and don’t have, and the confidence of the information (Koroorian)
- Beware of using the word “coverage”, rather say security, or risk (Birgisson)
- To me, coverage is a start of thinking about what to test (Bergqvist)
- If this isn’t reported “upwards”, you avoid the biggest risks (Andersson)
- The map and the creation of it is more important than the numbers on it (Öberg)
- It might look easy with a number, but it isn’t (Edgren)
- This weekend is evidence that standards don’t work. We can discuss forever, and that’s what is fun! (Emilsson)

These Lightning Talks (5 minutes including questions) were held:
* Martin Jansson on the tester’s greatest nemesis, the view that anyone, thus any idiot, can test.
* Henrik Emilsson on unjustified tests that open up to serendipity and new ideas.
* Robert Bergqvist had an experience report that showed that exploratory testing also needs planning.
* Azin Bergman on the common view that testers have lower status than other roles.
* Simon Morley on root cause analysis heuristic FICL: Framing, Information Gathering, Consensus, Learning.
* Rikard Edgren on Binary Disease – our tools have shaped way too computeresque theories.
* Christin Wiedemann on open-ended plans without fixed scope.
* Sigge Birgisson shared an experience report with developer collaboration under the radar.
* Steve Öberg led a brainstorm on testing analogies with story from the Aeneid about not following the Delphi Oracle.
* Ola Hyltén showed the Johari window that can enable better communication between leaders and team members.
* Torbjörn Ryber led a discussion about doing a context-driven conference in Sweden.

It is very easy to organize a workshop like this when you have 15 motivated, passionate testers.
You just need a venue with few distractions, a theme to focus, an agenda and discussion rules, plus food and drinks.

Looking forward to SWET3!

Once again I have spent the weekend with members of the cream of Swedish testers. This time The Test Eye trio consisting of Henrik Emilsson, Martin Jansson and Rikard Edgren were the hosts.

The theme was Exploratory Testing and Planning and we managed to keep the discussions within that scope most of the scheduled sessions. The informal test talk, music and drinking session from 18.30 to 03.30 encompassed a rich variety of discussions mostly outside that theme. Robert brought a fantastic instrument called the beat box (I think?), Rikard the guitar and the rest of us had guitar picks, maracas and music talent to some degree. The starting act was “The  Spice Song” composed and performed by multi talented philosopher-baker Rikard. Other highlights were “My Sharona” performed by many of us, later followed by interpretations of Cornelis Vreswijk and early in the morning sometime a try at “Bark at the moon”. The barking was considerable but I prefer the original version with Ozzys falsetto. And let us not forget the African freedom and tribal songs performed by the “missionary men”. Music is so much fun! If we ever start a band it will be called the Testicles.

Johan

But let us start with the conference pass. First session was Johan Jonasson from House of Test. He told us of the success he had with two young girls from help desk. I bet that sentence caught your attention! Well it was just as exciting as it sounds – it is all about testing. Jonas’ task was to manage the testing of a consumer product application and the resources he was assigned was two support people. He pointed out that they lacked education in IT and testing but were very motivated. He had too little time in order to prepare any detailed instructions. Test instructions consisted of some form of user stories created by himself and scenarios built by these. They tested together following the instructions and were coached by himself at a couple of meeting each day. They turned out to be great at taking notes and very good at finding problems. In the following open season we concluded that they did not lack education at all. They were used to taking support calls and analysing the caller’s problems so they had both product knowledge and experience on analysing and describing problems in text. Some reflections was that motivation is a very important factor and that less detail control of curios and motivated testers made them perform better. I can think of at least one client that uses support personnel but detailed script that should try the approach with a more exploratory format of working. As first sessions often do, it lasted for almost four hours and the discussions were never once uninteresting.

Tobbe

Number two on the list was myself with an experience report of testing in a loosely controlled and volatile project. Given the role as tester on the project I gradually moved into project management, requirement elicitation at the expense of much less testing than planned. Some of the things I did was to create a product backlog, organise weekly scrum meetings, create an effect map and a status graph. My overarching goal was that we actually deliver something to the customer and whatever needs there were somebody needed to take care of them. Success factors was that I was not only allowed but encouraged to assume new responsibilities by the other project members and that I find it interesting to take on other tasks than testing. The downside was that I had less time testing and, to be honest, less motivation to test. Maybe I would have tested better if I was given full-time on the project but when that was not the case I prioritised other tasks in order to move forward. It should be the testers goal to do whatever is needed to bring the project forward. I see similarities with the Scrum thoughts that we have less specialised roles. We had discussion on whether the artifacts I created was testing or not but the question is if it matters as long as they are important and they get done?

Fredrik

Fredrik Scheja of Sogeti was the next presenter. He told us about his success with an exploratory testing approach on a large system with frequent releases. Every tester assumes responsibility for analysing, test planning and execution of one or more items at a time. One of the dominating discussion in open season was the fact that he claimed that he built this approach on TMAP. Since most of the participants claim to be members of the context-driven school this is quite a daring statement to make at a peer conference. TMAP is clearly factory school which is the opposite of the context-driven approach. While I think none of us doubt the success of the work process used, many of us claim that it is not really TMAP-based just because you pick some parts that you find usable and then tweek them to fit what you really want to do. I think it was Johan that used the metaphor “Picking very few raisins out of a very large cake”. James Bach tweeted that “If you take a nice cake and drop it in a mud puddle, don´t bother with the raisins”.  Henke Andersson attacked the claim that the 12 step checklist for test planning really was the best to use for all planning in all situations and wanted to see a local adaption for the project that was very unTMAP(is there such a word, well now there is!). The discussion continued for another hour on Sunday morning for all of us except Ola Hyltén that by mistake(?) forgot to set his alarm when he went to bed already at 3.30. We thank you Ola for giving us an opportunity to make fun of you J

Lightning Talks and Conference plans

Last session Saturday was a number of lightning talks that were entertaining but to be honest I don´t remember much of the ten five minute talks. If someone else has a record or want to say something about them – be my guest. I do remember the last one where Henke told me to remind the group of our idea to arrange a context-driven conference in Sweden within the next year. All said it was a great idea and many wanted to help out. Our first suggestions was that we need a year to plan and arrange, it should take place in a larger city – probably Stockholm. Size could be 300 participants. Goal is not to make a fortune but not loosing money. Planning for a small profit gives us some room for unexpected costs. We would like to really focus on the context-driven community, only context-driven talks and not being sponsored by any certification organisation. We want to have some key players from the USA and certainly from the rest of the world as well. We may want to consider some tracks only in English and some in Swedish. All thoughts welcome.

Dinner

Dinner was a seafood buffet at the toll house by the sea. Since three of our four vegetarians have redefined fish and shellfish as vegetables they happily dug into the buckets of crabs, shrimp and langoustes (or whatever the English name for havskräfta is).

Martin said he was disappointed there was not a pool and agreed that he could possibly assume some responsibility for that fact since he was the one that booked. We focused on the music part instead as told before.

Saam

Sunday morning started out with another hour of open season on Fredrik followed by Saam that explained his goals to change testing and reporting on the large company where he works. We spent a couple of hours discussing green/yellow/red versus 1-5 versus happy faces and sad faces. The goal was to move from numbers – which say very little – to a more qualitative approach. This can be quite a challenge in an international and global organisation. We all look forward to learn about the results of Saams intentions in the future.

The happy ending

After some hugging, handshaking and some tears we left for the mainland. Yeah, cause I forgot to tell you we stayed at Chicken Island outside Gothenburg.

Henke Andersson has promised to arrange SWET3 in Malmö this fall. He mentioned that it will focus on ET standards and the need for ET certification…or maybe not. One subject that I would like to discuss more is teaching testing.

Additional info on Twitter #SWET2

The delegates were: Christin Wiedemann, Torbjörn Ryber, Azin Bergman, Fredrik Scheja, Henrik Andersson, Johan Jonasson, Ola Hyltén, Sigge Birgisson, Simon Morley, Rikard Edgren, Henrik Emilsson, Martin Jansson, Steve Öberg, Robert Bergqvist, Saam Koroorian.

Now and then you hear that developers should implement better support for testability, so testers can work more efficient.
This is all well, but what about the opposite; how can testers make developers go faster?

System testers have system (and a lot of other) knowledge, and we can see if the product turned out really useful.
We can find major problems, but also trivial ones that are easy to fix, and both are needed for ambitious projects.
We can find small Enhancements, nifty little additions that are fast to implement and test, and make the product better.
This can be called low-hanging fruit; we find them, and the developers pick them.
All you need is an environment that encourages looking at the product’s best before looking at the requirements and to-do-lists.

There are a lot of other ways developers and testers can help each other, think about it, come up with good ideas for your situation, and add a comment to this post!
The last time I helped developers finalizing some of their boring unit tests, it didn’t take long before I got a lot of energy in my direction.
Mutual interest and collaboration inspire each other; you spend some time, but get more back.

And to get back to the initial thought; it’s a compelling argument to suggest that logging of this and that will make developer debugging much faster.

I have entered EuroSTAR’s VideoSTAR competition for 2011, main reason might be that I want more people to see the excellent introduction movie Mårten and Henrik made for me a couple of years ago.

My title is “Set All Testers Free!”, and I can’t say I have the details set for a talk, but it will be about external freedom (allowed to test anything, your own control of deviations and environments) and internal freedom (liberate the minds of the testers, think bigger!)
We shouldn’t think or act like machines, since software is made for humans, by humans.

There are a lot of other interesting testing videos (especially the one on serendipity), so go there, have a look, and cast your vote, if you want to.

The big drawback and big advantage with reliability testing is that it is easiest and most effective to perform together with other testing. A separate automated reliability regression test suite could cost an awful lot to implement, but reliability in your spine when performing any type of manual test, together with deviations, is cheap, interesting, and powerful.

If you look at Reliability from a standards perspective, you will see a lot of measurement methods like Mean Time Between Failures. You don’t need to use these. You can test and find important information anyway.
The most lightweight method is to ask these questions to heavy users of the product:

Reliability. Does the product work well all the time?

- Stability. Are you experiencing (un)reproducible crashes?
- Robustness. Are there any parts of the product that are fragile and have problems with mis-configurations or corner cases?
- Recoverability. Is it possible/easy to recover after (provoked) fatal errors?
- Resource Usage. What does the CPU, RAM, disk drive et.al. usage look like?
- Data Integrity. Are all sorts of data kept intact in the system?
- Safety. Is it possible to destroy something by (mis)usage of the system?
- Disaster Recovery. What if something really, really bad happens?
- Trustworthiness. Do you feel you can trust the system?

You may also want to perform some specific tests aiming at the different sub-categories.

Stability. Run the product for a long time, without restarts.
Automate a simplistic scenario and run it thousands of times in a sequence.
Count the number of non-reproducible crashes per day that happens to your team.
Try really hard to reproduce the non-reproducible issues.

Robustness. Provoking error messages is fun, and don’t forget to check spelling and if error message helps.
When this is important: hit hard, hit many times.

Recoverability. Turn off the power for machines performing important things, restart and look at behavior.
Whenever an error occurs, try to recover, and consider if it is easy and intuitive.

Resource Usage. Look at system resources now and then.
Stress the system in various ways (but only spend time on this if the project is interested in results.)

Data Integrity. Use all types of data (numeric, strings, out-of-range, invalid, empty, Unicode), in different sizes (small, medium, large) on different systems (localized OS, regional settings, different fonts) through all parts of the system.

Safety. Do thorough brainstorming around scenarios where people can get hurt. Be aware that ambiguous or missing information can be very dangerous if they affect important decisions.

Disaster Recovery. You probably don’t want to test this for real. But you can ask developers or others if there are possibilities of continuing using the software after a crucial machine has disappeared. This is one of those characteristics that either is irrelevant, or very important.

Trustworthiness. Note down all inconsistencies in behavior, or moments when you are unsure what the product is up to.
Tell the project how you feel about the product’s reliability.

The best start to get reliable software is to have really solid code, and properly customized code checker tools can help you with this.

I guess the list could be a bit longer and still be lightweight; feel free to help me out!

I spend a lot of time testing new features for the next release.
I actively try to not test the features in isolation, to not use the easiest data and environment.
One example of this is that I often use “documents” that are more complex than necessary, that includes elements and strange things that aren’t obviously related to the functionality in focus.
E.g. if I were to test text coloring in Word on a fully-fledged book with different types of footnotes, a lot of formatting and images et.al.
This doesn’t cost much time, but now and then expose weaknesses, either in the new functionality, or in the old product.
I guess many do this, and I recommend it to anyone who has the freedom or guts to control their test environment.
Maybe the name Background Complexity Heuristic can help you remember it.

When a test is “completed”, I like to do something more with the product, preferably something error-prone, popular or the next thing a user might do, not necessarily related to the new feature.
I don’t think too much, rather just press F1 if I can’t think of anything better, since I don’t want this extra-testing to take too much time.
I call this the Do One More Thing Heuristic.
It helps you learn, and find problems.

For both of these tricks, it might take some time to pinpoint problems, but the alternative might be to not know about it at all.

There are some discussions about current certification schemes, but there is not so much attacks and defense of the actual content.
This is from ISTQB Glossary 2.1:

black box test design technique: Procedure to derive and/or select test cases based on an analysis of the specification, either functional or non-functional, of a component or system without reference to its internal structure.

A first start of the narrowness I dislike is “test cases”.
Why must test design have a set of instructions and expected results?
I think test design can have many forms: detailed, visual, one-liners, tables, un-documented, charters, and it is not good to steer testers towards a limiting format, that in my opinion seldom is appropriate (because it stifles serendipity, promotes confirmation bias, is cumbersome to review, and time-consuming to write and maintain.)

“Analysis” might be the most under-estimated and un-elaborated areas in software testing.
In ISTQB foundation its allotted time is about 2 minutes, and I haven’t seen any interesting on this in the Advanced or Expert syllabi either.

Maybe it is because the test basis only consists of “specification”. I know it happens that tests only stem from specifications, but I can’t understand why.
Don’t we want to find out how the system really behaves?
Do we genuinely believe that the writers captured everything that might be important?
Are we consciously neglecting everything we learn throughout the development project?
Requirements are a good start, but there are a lot more to look at.

Have they written “derive and/or select” to make sure that no creativity and new ideas appear in test design?

That the definition reads “the specification” is a symptom of the un-holistic world view that each function/feature should be tested in isolation.

At least there is mention of “non-functional”, but I don’t want to detail my critique on their view on this (I think it should be done together with other testing, that it doesn’t have to be done by experts, that it is OK that it isn’t measurable in quantitative format, that testers should have a broader view and knowledge.)

I haven’t taken the ISTQB training, with the right teacher it might be great, especially for newcomers that want a glimpse on many aspects of what software testing is about.
But it is a pity that the content is so meek, bleak, weak.

If you haven’t done it before, have a look at the Software Quality Characteristics that we published last year: TheTestEye – Software Quality Characteristics

You can probably imagine ways of testing for all of these quality characteristics yourself, and you might even come up with good oracles that can assist you in the interpretation of the test results. However, the Charisma characteristics might be the most subjective ones; and it might also be a case of where it is really important to find out, for each quality characteristic respectively, which stakeholder whose values matters the most.

So how do you test for a product’s Charisma?

Charisma. Does the product have “it”?

• Satisfaction: how does it feel after using the product?
• Professionalism: does the product have the appropriate flair of professionalism and feel fit for purpose?
• Attractiveness: are all types of aspects of the product “good-looking”?
• Curiosity: will users get interested and try out what they can do with the product?
• Entrancement: do users get hooked, have fun, in a flow, and fully engaged when using the product?
• Hype: does the product use too much or too little of the latest and greatest technologies/ideas?
• Expectancy: the product exceeds expectations and meets the needs you didn’t know you had.
• Attitude: do the product and its information have the right attitude and speak to you with the right language and style?
• Directness: are (first) impressions impressive?
• Story: are there compelling stories about the product’s inception, construction or usage?

All of these are somewhat intuitive and I guess that you, consciously or unconsciously, test for these everyday in your project. You can also focus on testing these in the same fashion as when testing for other quality characteristics.
However, if (some of) these software quality characteristics really matter in your project and are core values for your product, you better question your own capability of being able to test this.
By question yourself; you might realize that your oracles aren’t powerful enough in order to really being able to answer the question “Is there a problem here”.
You might be too biased; or you might not be able to really understand what the stakeholders need or value. Or you might simply be too unimportant in order to even “have opinions”…

One way of addressing this is by letting a proxy user act as observer and interpreter of the test result. This can be a good approach when testing for software quality characteristics where you suspect that your own oracles won’t be enough in order to understand what is important. Well this isn’t news for many of you, since user testing has been used successfully for years. But my point is that you can spend some extra thoughts on which you select as your proxy oracle – with a mission to test for those selected Charisma values.

You can do this kind of testing by letting a proxy sit next to you while you test; or by letting users test the program themselves and you interviewing them afterwards. Another way can be to test or demo in front of an audience of proxies that observe and interpret the result as you go.

The tests could be designed as open questions that you would have asked the program and watched it respond, but instead you ask the questions to the proxies and letting them answer in the way they interpret the result.

In a recent discussion with Henrik Andersson on Twitter regarding some consultancies being or claiming to be best at testing. Here is the initial conversation:

Henrik: Dear consultant companies why are you calling yourself consultant when all you talk about are recourses and invoiced hours. Shame on you!

Henrik: Many “consultant” companies claim to be “best at test”. You only train your testers in ISTQB and maybe a TMap book. That can’t be the best!

Henrik: All you companies that claim to be the best. I suggest you to have a show down to actually demonstrate if any of you are any good at all.

Martin: I take you on that challenge!

Henrik: are you one of those companies that claim to be “the best”? I have never heard you say that. But I’m always up for a game

Martin: No, to claim that is ignorant. But we are good at test related tasks. How do we measure ourselves then?

Henrik: you measure yourself by sharing your knowledge, demonstrate your skills, discussions with peers, trying new stuff, failing.

Martin: Yes, I know. But can you really compare?

Henrik: you can sort out those who sucks at it. but more important is not comparing but gaining respect and recognition from peers

Henrik: for me it is not about having one winner. it is about contributing to do my part in developing our community & learn from it

Martin: If you are good it is harder to distinguish who is the better as tester. I agree with you regarding the community etc.

Martin: I mean, would you compare which information was the most valuable between two who competed? I think it is hard.

Martin: Still, I see some testers who I think are really great while others are just good. What criteria do I have then?

There are many testers who have great renown, thus they talk a lot about testing. I’ve seen a handful of them show their skill. So they fit in with the tester Henrik mentions:

- by sharing your knowledge

- demonstrate your skills

- discussions with peers

- trying new stuff

- failing

A great tester might be an introvert that do not discuss with his/her peers or share his/her knowledge. Is it even required to demonstrate your skills? Are we talking a bout a renown tester here or an unknown tester who is just great in secret? Still, in order to be hired based on your reputation or renown you might need to fulfill the criteria that Henrik lists above. A great tester is only great in certain contexts, finding someone who is great in all contexts would be very rare. Being a generalist and general systems follower would mean that you could adapt to most situations, but would you be “great” in all of them? A specialist in one domain with great testing skills might be better than the generalist? Highly likely.

One criteria that I think is important is that a tester should be different from the next tester. This does not really apply to the tester that goes solo or prefers working alone. If we instead consider having a tester in a group that is really great at some aspect in testing. He/she is not great at everything and will not be able to handle every situation, but as a group they are well equipped to handle many, many contexts.

In my test team I have many great testers, as I see it:

- All have a high focus on what is valuable to the stakeholders.

- One loves everything that is complex and hard to understand, digging in to see what it is made of.

- One is totally unafraid, young and goes paths senior people would not go.

- One is extremely creative and comes up with tests that breaks everything in exciting new ways.

- One has an affinity for making everyone else feel better, thus making everyone work a bit better as a group.

- Some I’ve worked with a long time and probably knows things that I would miss,  thus covering behind me and I behind them.

- Some have the expertise in the current domain.

- Some have similar background and vocabulary with each other.

- Some are new to the group.

- Some are men and some are women.

- Some have excellent programmer skills.

- Some have experience from other roles in product development.

- Some are used to talking to the customer.

- Some are used to leading the group.

- Some are natural leaders.

- Some have worked together for nearly 15 years and still like it.

- Some have known each other for more than 30 years and are still best friends.

If you look at each one as an individual you would see one side, but if you look at the team and what they can acomplish you see greatness. Alone they would not be able to solve any situation in testing, but as a team they have a better chance.

This view is supported by the one that Cem Kaner writes about in his “Recruiting software testers” [1]. For a specific company he identifies an team that he would think he ideal. He lists it like this:

• Senior tester or test manager with experience in business operations or human resources. This person has worn the shoes of the customer for this system. For a vertical application, I think this is essential.
• Senior tester or test manager with strong test planning skills. If this is the test manager, she needs excellent mentoring skills, because she won’t have time to write the test documentation unless she is an individual contributor.
• Test automation hotshot, willing to serve as the group’s tool builder.
• Talented exploratory / intuitive tester, someone who is really good at finding bugs by playing with the product.
• Network administrator. This person has the dual role of helping the other testers set up and deal with the ever-changing configurations that they have to test under, and designing configuration tests to determine whether the product will run on most of the systems in use by the product’s customers.
• Attorney who is willing to wander through the various statutes and regulations looking for rules that the program must cover.

When you assemble a test team in product development, do you then consider the above traits and properties. So, back to the question that Henrik asked initially about consultancies stating they are the best in testing. I am sure they have good testers as individuals, but can they match a team that is handpicked to complement each other? I hardly think so.

When project managers need more resources for the project and especially testers. Do they then ask for “I want a tester” and “he/she must be ISTQB certified”. Being certified in testing is a story of itself, which I think is disturbing.

First, we really need to consider how they come to the conclusion that they need X number of testers. Was that the budget talking or was it someone who did the estimate on how many you “exactly” need? Ok, let’s assume it was budget to simplify the discussion somewhat. How do you then go about assembling the test group? Do you consider things things that I listed above or do you just want a tester, no interest in what they know and how they would fit in with the rest of the group. If they compliment each other is not a priority?

- What if you placed someone who is a (only english-speaking) great tester in another group of other testers who do not know english at all?

- What if you placed a collection of great testers, where each came from a competing consultancies?

- What if you only had introverts as testers in the group?

- What if you only had extroverts as testers in the group?

I often see the requirement in tester ads for domain knowledge rather than knowing anything about testing. I think domain knowledge is important, but you probably need at least one with it in the test group. There are so many other characteristics and backgrounds you want from people in the group.

So, my conclusion is that being a great tester is very context-dependent and you cannot say that you are the best.

References:

[1] Recruting software testers –  http://www.kaner.com/pdfs/JobsRev6.pdf