thoughts from the test eye

Yep, as the saying goes in the Scuba diving world (I’m an ex-instructor)…

“Certified doesn’t always mean Qualified!”

I agree, the certifications are becoming a joke and being run as a money mill. I’ve held the CSTE from QAI twice and have let it lapse. Never did anything for me other than allow me to add 4 letters to my resume. With 20+ years of software testing experience my certification came from the “school of hard knocks”.

Guys, CSTE is not a qualification however it shows the professionalism in you which doesnt mean that who doesnt possess CSTE is not!! 🙂 Still a certification shows the in depth knowledge we have on any particular subject say here Testing.. Many have so much of experience still if you ask a basic question like ‘What is a quality?’ defineltely no one will have a proper definition!! Think!!

I do agree with Henrik.
Though certification not a constraint for hiring candidates in Asia, people are giving a lot importance to certifications. I know anyone not from testing background can solve these questions, so you can’t judge him/her on basis of this certification.

If employers can’t deny the importance of these certification at least don’t make it as entry criteria for hiring candidates.

Will soon raise voice on this topic on my blog as I am nowadays getting too many queries from readers about this certification guide. Thanks Henrik for raising this point.

PS: I don’t have any of these certifications 😉

@Jim and Vijay.
Thanks, and continue to spread the message! 🙂

@Maya.
I don’t know anything about CSTE, I was rather talking about the ISTQB certification. If the CSTE certification is based on practical and theoretical skill, then it might be a good certification!?

First of all, very nice and interesting blog you guys got here!

Regarding this post. Interesting topic. And, as you mention, there is debate ongoing. However, I am not convinced the ISTQB certification is to blame althoguh the ambition level of the certification could certainly be raised a fair bit for the foundation level.

You write: “So what it really means is that a person that hold this certification has managed to get at least 30 multiple-choice questions right out of 40”.

Yes, exactly. In the same way a univeristy degree means a person did something. A diploma means a person did something. 10+ years of experience in SW testing means a person did something. None of the above means he or she is a good tester.

As I first understood your post, you want to make people in hiring positions aware of this fact. I agree this is good.

But you end by stating “Let us treat software testing as a serious profession and throw out this certification as soon as possible”

My quesation here is: are you attacking the content of the certification, or the existence of certifications? Where you not attacking the usage of the cerification by the hiring managers?

Anyway, if you are attacking the content then I say great – that debate must always be onging. If you are attacking certifications in general then I ask you if you provide any alternatives? If you are attacking the usage then I certainly agree with you as I stated above – but my opinion is that, bottom-line, it is up to the user of information to decide how to act on certain information.

If the reality is as Jim descries it in his comment above, that the low requirements on passing the certification is to enable a “money mill” then there is a strong need to intensify the debate even further. And your blog post becomes ever so important! But please let us be clear on what problem we are attacking and also if we provide any alternatives or not.

@Saam.
Thanks for your comments!

Let me try to be more explicit: I am not attacking certifications in general, I am attacking the ISTQB-certification and its content.
I do worry about people getting rejected because not having the ISTQB-certification.
Maybe there are other certifications that are worthless, maybe there are some that might be useful and valid.

————————

When I have been involved in recruiting testers for projects, I have received applications where they have been presented as software testers but the only evidence of any experience in software testing is that they have been certified by the ISTQB in Software Testing. And the companies that have presented those testers have honestly said that they believe that the persons are suitable because “they are certified software testers, what is your problem?”.

——————

There is much to say about the alternatives, so I will explain what I see as viable and better alternatives to the ISTQB-certification in a separate post. Hope that you will pop in and read that!

Thanks for clearification, I see the problem. “certified software tester” is rahter misleading! Maybe then the “name” is what is actually causing the problem. If it was called something like “certificate of basic knowledge in test terminology and concepts” you might not consider it _neccessary_ to “throw out this certification”.

““they are certified software testers, what is your problem?””
That is hilarious! Or actually it is quite scary… Either way, I certainly see the need to make clear the facts around the ISTQB certification.

I am looking forward to reading your next blog post!

I have talked to recruiters while looking for work and the first thing I am asked is if I have the Foundation certification, nothing about my experience, etc.

Recruiters seem to use the ISEB as a gauge for a tester (or are asked to by their client to) yet a lot of them are not aware of the Intermediate certification, I frequently see job descriptions which mention Foundation and Practioner only.
So even though they are using these exams to help them judge my worth as a tester they are not actually aware of all the certifications available.

Here’s a post regarding skilled testers, certification is mentioned: http://tinyurl.com/yemvbgj

@Tony.
Thanks for your comment!
I have read your post and agree to some extent.
———————–
Regarding the analogy with a driving license, I just want to add that in Sweden there are two tests you need to pass in order to get your driving license:
First a theory test and if you pass that you do a driving test. I.e., both theory and practice.
In order to pass both tests you often need to go to driving school in order to study theory and practice on actual driving under supervision.
When doing the driving test you have one inspector/supervisor sitting next to you which is there to judge whether you pass the driving test or not.
Even so, no one would dare to say that a person that just have taken their driving license is particular good at driving a car…

Why do you think there is a preassure to have certification such as ISTQB? What are managers actually ask for?

I believe the original idea behind this has been lost and instead it has been replaced by what we see today.

I remember in my early test career that me and my tester co-workers discussed a lot about terminology, methods and such. We did not agree on the simplest things. In some cases management said “Just decide what you mean!”. If I had been manager at that time I would have thought that some kind of common understanding would have been fine.

But as it is today seeing that a skilled tester is the same as someone who has been certified is utterly wrong.

Still, I experience almost every day situations where someone uses a term or a concept without knowing (as I see it) the full meaning what he/she says. In those cases it would have been nice to point at a common dictionary or terminology list. Is ISTQB filling that void?

@Martin, thats exactly how I have viewed the ISTQB certification! And yes, I believe ISTQB does fill that void, as it is providing a common starting-point for testers (and anyone else how wants a basic set of terms and concepts). It is important that the syllabus is continously updated as the industry is not standing still. However, it is not of the highest importance that we all agree on everything it includes. I have read that “Bill Curtis has said that in a room full of expert software designers, if any two agree, that’s a majority!”, and I think we can assume the same goes for testers. However, it has become clear to me (thanks to this blog post!) that the certification is being severely missused by the industry. I suggested in my comment above that a simple action as a name-change might help this situation, as I still think ISTQB fullfills a meaning.

Maybe it would all be well if the certificate stated “I know one way of interpreting some of the terminology used in software testing”?

I don’t have a certificate myself, and if a company required me to have one, I wouldn’t wanna work for them.
But, everyone don’t have that option…

@Saam and Martin.
If ISTQB were to fill that void, they must first recognize and communicate what Rikard suggests, namely that the syllabus is “one way of interpreting some of the terminology used in software testing”. That is not gonna happen…
In my opinion, the terminology in the ISTQB syllabus is out of time, context-free and challenged and it is probably never going to meet the expectations of all people in our business.
———————

@Martin.
Regarding terminology.
I think that the problem with terminology that you mention might in fact be caused by the existence of (not lack of) a common terminology dictionary. E.g., the terminology promoted in the ISTQB syllabus. It is one thing to use and point to a terminology list (especially if you haven’t reflected upon the meaning of it); and it is a completely different thing to question and discuss the terminology in order to understand what it means in your context.

I think it all comes down to what one expects from the ISTQB certificatation. And obviously there are people out there who expects far too much. I can agree with you, it is a certificate that states “I know one way of interpreting some of the terminology used in software testing”, nothing more, nothing less. Still (however unfortunate it might sound) I see a real need in the industry for just that, and therefore I see a need for the ISTQB certificate.

Do I dare to ask what you think of an introduction of a testing standard? (ISO/IEC 29119, http://softwaretestingstandard.org/ ) 🙂

@Saam.
Well, I do not believe in a testing standard such as the one you provided. Simply because I do not think that you can standardize something that evolves every day and is still under development.
And since software testing is a social science (http://www.testingeducation.org/a/ifipkaner.pdf http://www.kaner.com/pdfs/KanerSocialScienceSTEP.pdf ) and still evolving, I do not see a purpose of having a standard that has its starting point in the process corner: “The aim of this project is to produce one international standard for software testing that covers the entire software testing lifecycle, throughout the analysis, design, development and maintenance of any software product or system“.

Oh I forgot the most important thing: A standard implies that there is a preferred way of doing things, i.e. a “best practice” . I do not believe in such things as a “best practice”. I believe that software testing should be context-driven. http://www.context-driven-testing.com/

Yepp, kind of suspected that (read the about info previously…), but dont worry, you might still find parts of it useful: “To the context-driven tester, the standard provides implementation-level suggestions rather than prescriptions.”. 😉 And, I must admit, I do not believe in a standard either. However, I see some other benefits with it, especially when trying to influence/change some context (ultimately, influnce/change “what we get”) it is a lot easier to establish commitment if you can point towards success stories or e.g. an industry standard to back up your suggestions. As unfortunately sometimes to “explain tradeoffs” just isnt enough.

That sounds interesting. 🙂
Still, the problem is if you point to some industry standard, how can you be sure that it would be appropriate in your context? And if it happens to be appropriate for your context (it might well be so), how can it be appropriate in another context (lets imagine that you switch to another company/client/project)?
————
I am also curious, could you elaborate some more on how you think you can use the standard to “influence/change some context”?
————
If you are interested in asking questions and getting feedback and tips regarding how to do in your context, you could join the context-driven group and discuss with some really good people. http://groups.yahoo.com/group/software-testing/
———–
Excuse me for putting a lot of links in my answers… 🙂

Firstly, no problem, I love the links, keep them coming! 🙂 Ok, I’ll try to explain how I see this. One can either adopt to the environment or try to change the environment. Or both. As I interpret “Ultimately, context-driven testing is about doing the best we can with what we get” it is about adopting to the environment, please correct me if I am wrong. I think this is the best starting point, one needs to adopt. However, if you work in the same type of environment for a longer period (same type of product, projects, people) you might start to question certain things. Why do we get “what we get”? We might see a possibility of incresed effeciency if we would be able to apply a different type of approach but it would not be effecient in our current context. Lets say this is due to some corporate requirement or due to the lack of a certain type of information. If we could achieve a change in our environment that would enable that type of information or discard that requirement then we could switch our approach and hopefully become more efficient. I consider this as changing our environment rather than adopting to it, and in order to achieve this change we must get people commited. To do this we can show them our anticipated increase in effeciency. But to fully win them over we might need to also show them that the approach we want to enable is a proven concept as it is considered industry standard. (Even if it is not proven in our context or the slightly modified context we want to create.) I am not sure it became more clear?? And also, it is probably not a very good reason for creating a standard…

OK, I think I see what you mean.

—
Changing the environment is sometimes a hard thing to do, and it is sometimes even impossible. But it is not completely impossible! 🙂
In order to prove that your new approach is a more efficient way I think that you need to investigate what kind of service you should provide as a tester in your project. One start is to investigate what kind of information the stakeholders are interested in, and come up with ways to address that mission. If you then find a more efficient way to elicit that information other than what you do today, you now have an incentive and a motivation to do it. If you are patience and start in a small scale, you might reach far after a while.
Another thing to do is to question the company requirements and ask why and if parts of “what we get” are really needed. This way it might become obvious for the executives that the current way is not the preferred way. But be prepared to fight some battles… When you question those type of requirements in a company that has a long history, you are attacking the company culture; you need to be aware of that people might have built their careers on this culture and others have been hired and adopted to roles that they hardly want to get rid of. 🙂
—
As you mention it is really important to get people committed, but I have rarely seen that work by first implementing an approach and then try to convince and persuade about the efficiency. Simply because of politics and the fact that people have a hard time admitting that they are wrong (me included)… Instead, my experience is that you need to get a commitment on the mission first, then design an approach that meets that mission. This way you act in the spirit of the executives since they have been with you all the way.
—
If you still want to point to an “industry standard”, you need instead to point to a practice that has been good or promising in a similar environment and context as yours. To me it looks like it would be impossible to describe an “industry standard” that would be suitable for the full range of contexts that our “industry” covers (if we even could say that we are an industry!?). To illustrate what I mean I could mention a few very different products that would be hard to develop using the same test approach (or a standard approach): pace-maker, web site, open source plugin to WordPress, financial/banking system, airplane, Microsoft Word, web browser, compiler, cell phone, booking system. And have in mind that the context might differ between companies/organizations that develop the same type of products listed above.

Thank you for thorough feedback to my comments! Yes, as I see it, even if it is hard to change the environment I thing the long-term-vision should always include that, at least as an option to explore. But first of all one needs to adopt to the current environment. I agree with you regarding the ways to go about getting commitment and the need to point to an “industry standard” is limited. However, sometimes you tend to get peoples attention when you include references to e.g. IEEE, ISO, or e.g. CMMI or other similar entities (that the reciever of your message knows and respects) as they percieve it as there is a certian level of “confirmed” professionalism and weight behind your words. Thats my experience.

@Saam,
I find it strange that it is so common that test experts within an organisation must point to industry standards, IEEE, ISO, CMMi and consultants in order to get the attention. I often wave Cem Kaner (as professor) in peoples eyes to get the attention, but it is usually a last resort. I recommend that you get the consultants behind you, to support what you want so that it is clear that it is you that have the driving seat.

I think resistance to change is overrated. If you work by example before writing the process or document there is a bigger chance you will succeed, as I see it. This will win those who will be affected by the new way of working. Still, documenting how you do work in order to make it easier for everyone to understand is best in the end.

I agree with you Martin. And, as you say, the resistance to change is maybe not the problem. However, change on a higher level still needs the approval and enabling by management (money and prioritization power) and for a decision to be made people often want “some kind of insurance” it will work.

There’s an aspect that hasn’t been mentioned yet, and that is that growing attempt at collabrative team environments in which case, we want to be using language everybody understands.
Our we going to try make everybody speak our language or speak in a language everybody understands?

@Tony.
I am not sure I understand what you mean. Could you try to clarify a little bit?

One of the BIGGEST problem with the certification as of ISTQB is that in order to be able to cover so many subjects in such a short time 2,5 days – matters are so simplified that they become ridiculous.

Look at boundary value testing – it is taught in such a rote manner that people are made to believe that you need to find the obvious simple partitions and then you take the least number and one below etc – which is hardly learning how to test.

State graphs are also taught in such a manner that you always start with a simpe table or a graph and look at what transitions are possible. The hard thing is to create the table or graph and the ones I create are almost always updated continually during test execution.

Having a common vocabulary would be OK if it was a decent – not perfect one – but really bad if it it stops people from learning. It is also different from teaching poeple what is best and second best when it comes to solving certain problems. Take a look at a practice exam question and try to find situations where ANY of the four alternatives are valid – it is not that hard. To say that one answer is better than another is to totally forget the idea of context. I recommend Jery Weinbergs Exploring Requirements for anyone that wants to learn more on meaning and context.

I am certified at ISEB practitioner level since 2004 – that has since been reduced to simple multiple choice questions because to few people passed it. Lack of skill or to hard to grade when thining of context? I have had to de-learn a lot of test theory since then.

@Henrik, Apologies for the delay. What I mean is that there are comments here that talk about standards, language standards, etc. We all know that certain phrases mean different things to different people and organisations which can sometimes cause confusion. However being aware of it should mean that we are clarifying what is meant.
There is however a……….I’ll call it a growing movement to bring in more people who are not testers, not developers but users, BA’s, etc and bring in them as part of the team. Things like agile and BDD and these people who will be working with us will not be aware of the industry ‘confusion’ so perhaps instead of trying to sort out that confusion there should be a move to simplify language used (as BDD does) so that everybody (including non testers and developers) understands.

@Tony. No problems with the delay! 🙂
I hear you on this. The issue with what you propose, as I see it, is that in order get everybody to understand your language you need to be able to explain what words mean. And in order to explain what they mean you have to deal with the fact that there are disputes regarding the meaning of certain terms, methods, etc. I.e. in order to present a [simplified] language you need to decide upon which interpretation that is valid; and then we are back on square one again.
I do not think that simplifying is a solution. Instead we should realize that it is more complex than what could be taught on a two and a half day course.
Of course it would be nice if we could speak a language that everybody understand, but see what happened with Esperanto that was partly developed for the same reason… 🙂