thoughts from the test eye

Rikard,

A thought experiment:

Suppose you took the definition (implicit or otherwise) of “analysis”, “specification” and “test cases”, as derived from the ISTQB syllabus.

Then from this you were able to scope the range of testing ideas (types and grade of creativity) that would be developed from these guiding principles – let’s call that your “test ideas or creativity set” as guided by the syllabus.

Now, think about the areas that have a risk of being excluded or included due to the guiding of the syllabus – you noted quite a few (but that’s not an exhaustive list, of course) – biases, issues with reviewing ideas, no opening for serendipity, etc, etc.

So, one could hypothesize that a tester not directed by the syllabus could extend their test toolset by being aware (or learning about them) of biases, different forms of representation of ideas (or problems and issues) to aid anlysis and communication and a whole range of thinking aids for problem analysis and creativity.

So, given the toolset available for any given problem (test related) is it better to be constrained or not constrained in ones thinking?

I suspect the major issue with testing (or certification) is the awareness of the toolset that is appropriate for the testing problem (from both the tester and stakeholder perspective) – and inded, knowing when ones toolset needs refinement and improvement – and this is continuous.

The awareness is partly a PR problem.

Mmm, ok, back to learning…

I couldn’t agree more!
I would even say that the sentence “Procedure to derive and/or select test cases based on an analysis of the specification” is wrong since it is so narrow.

BTW, is there any definition of a test case that includes a set of instructions and an expected result?
A test case is an instance of a particular situation that you want to test. Right? But I am not sure the ISTQB wanna tell us that.

For me the “procedure” in the sentence I mentioned above might have test cases as input. I.e., a test design technique might assist you in developing/refining a test case or test idea into something more thorough – which might include a new set of test cases, test scripts or a more refined list of test ideas.

And I have taken the training, with a good teacher.
But I’m pretty sure that I didn’t get a a glimpse on many aspects of what software testing is about.

We can find faults in anything (especially as testers 🙂 ),
But that’s a useless effort.
Instead – suggest better terms that may be widely accepted, and make a change.

Simon, your comment make me realize that if speed is the most important factor, the ISTQB definition is pretty good. Fast to perform, and it still looks like a real testing effort (James Bach use the term “fake testing”.)
If something important slips through, we can blame the requirements.

Henrik, from ISTQB Glossary 2.1:
test case: A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]

Kobi, I don’t think it is useless to highlight important problems (which I think this example is.)
I don’t have a good short definition of “black box test design technique”, I don’t think we need it (no need for “black box” and “technique”.)
In general, I try to write “for” things, but the occasional “against” blog post is inevitable…
I have things to tell about test design, hopefully I will have something readworthy later this year…

Hi Rikard,
Nice blogpost, although I don’t (fully) agree with you.

The ISTQB content is derived from international standards and an extensive list of literature; both from the context-driven school as the ‘traditionalists’. It isn’t something the people of ISTQB make up in committees or something. The glossary is intended to make (some) agreement on terms so that a broader audience can understand and have a mutual understanding.
Standards (and their glossaries) are there as guidelines, not as set law. If you want to use a different definition you are (naturally) free to do so; but it could be wise to make a description that provides mutual understanding in your audience.

The term you chose about black box testing technique is also a nice one to discuss (as I’m sure more in the glossary are). I find your analysis of the term even more interesting, I would not have come up with this interpretation myself. THIS if find the most intriguing; for apparently the term causes ambiguous understanding and it was meant not to cause discussion. But I must point out that both thoughts we might have on this term are both interpreted within the definition and thus makes it what a guideline should do; give guidance. The important thing is to create a mutual understanding and make clear to your audience HOW you’re applying and interpreting things.

You mention ‘specification’ for example; this is also in the glossary.
specification: A document that specifies, ideally in a complete, precise and verifiable manner, the requirements, design, behavior, or other characteristics of a component or system, and, often, the procedures for determining whether these provisions have been satisfied. [After IEEE 610]

Again – not from ISTQB, but from IEEE 610 – I find this definition one to really discuss. A document? How many times have one of you ever been in a project were there was no document at all, but you had to use mails, notes and interviews to get to the information needed. SO I would not have used ‘document’ but something like ‘Information’.

I’m also wondering where you got the 2 minutes time from for analysis; my class did a whole bunch of exercises to understand the importance of analysis on source information about your SUT. Again; this is own perception and nothing to account ISTQB for, but more your course provider or teacher (or if you did learning on your own… yourself)

Next you mention ‘test cases’. This is also in the glossary (again not from ISTQB themselves but from IEEE 610)
test case: A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]

I’m finding this one still applicable although you might want to interpret it in the context you’re testing in. Even in context-driven testing you have a inputs and execution preconditions, although they are more sentient and less academic set; hence the craftsmanship needed. You also still have an objective and the ‘particular program path’ is stated as FOR EXAMPLE as is the ‘compliance with a specific requirement’ ; it’s not a rule to exercise this program path or to comply with a specific requirement. I would have been worried if there was a whole description on how to do this exactly and that you should keep to this procedure and rules to comply to the standard, but that isn’t the case.

Derive and/or select is for me an invitation to BE creative in stead of limiting it. A procedure to do so, could also be read as ‘A way to get to test cases’ or even ‘A way to write a teststory’. I see no limitations there. Why should this prevent new fresh ideas in a design? I don’t see this at all.

Further more, you start with a definition of ‘technique’ (action) and your first argument is about ‘design’ (artifact). But I agree on your statement that
a test design can have many forms: detailed, visual, one-liners, tables, un-documented, charters. And I still don’t see where there is stated that this mustn’t be done or shouldn’t be used. Even in your mind you set up ideas of what you should test and you have an expectation (else you wouldn’t be triggered when you see something that you could consider a defect/issue or bug (or whatever term you like to use).

That’s my two (well lot of) cents. I think my point is that no matter what basis you use, if it is contextual or ‘traditional’ or whatever; you should not limit yourself on what is ‘written’ but you should just test in the best way you can or what works best for you for that particular time, system, moment and need. I think that having this whole discussion focused on ‘what is said’ or ‘what is exactly stated’ is one that can go on forever and ever because there always will be different perceptions on what it should mean (no cook can cook for all tastes in one meal) , it’s better to invest that kind of energy in making yourself a better tester and explore (and teach) those methods, lessons etc. that work for you and that you’re happy with instead of giving attention to that you don’t agree with.

Hi Natalie

Thanks for taking the time to comment.
My view isn’t very benevolent, but your understanding shows that there definitely is room for improvements in the definition.
The reason I invested time in this example was that I gave a guest lecture on a school that were being taught ISTQB, to make them aware of controversies.

I have never been in a project where “there was no document at all”.
I have never been in a project where documents were the only source of relevant information.
There are probably always important information that can’t be phrased in “complete, precise and verifiable manner”, and this is natural: we are dealing with software made for humans, by humans.

The 2 minutes is a rough estimation; the syllabus state (at least) 15 minutes for a section where a seventh of the text was about analysis.

Sometimes I use “inputs and execution preconditions”, sometimes not. Again, should we have guidelines that limit our testing efforts like this?

I have been a sucker for semantics, and “derive” means that it comes from the source, that you don’t introduce new things.
But I’m glad that your interpretation allows new fresh ideas.

Regarding ‘technique’ (action) / ‘design’ (artifact), I see design also as an activity.
But I confess that I mix two definitions, and that I could have mentioned that ISTQB includes error-guessing, experience-based and exploratory testing.
I guess these are outside the “techniques”, but still part of the “test design”.
However, these activities does not get the proper attention, and are seen as worse since they don’t give proper coverage numbers (I will definitely not go into details here…)

I am not a context-driven member, I agree that “Standards (and their glossaries) are there as guidelines”, but I want them to be GOOD guidelines.

Good discussion and an impressive response by Nathalie, even though I don’t agree with everything.
I’m though annoyed by the fact that Nathalie uses the term ‘context driven’ as an interpretation of what I would sort under ‘exloratory testing’,
“Even in context-driven testing you have a inputs and execution preconditions, although they are more sentient and less academic set; hence the craftsmanship needed.”

My interpretation of context driven is almost exactly what is then presented in the last sentence.
“…. no matter what basis you use, if it is contextual or ‘traditional’ or whatever; you should not limit yourself on what is ‘written’ but you should just test in the best way you can or what works best for you for that particular time, system, moment and need.”

I think Nathalie did a sloppy generalization there..

Rikard, like you say certification or not is one discussion but what is really interesting is the content itself. I find it really hard to come up with good definition of terms in a small group and I find that the larger the group is the more difficult it becomes to agree.

What is even more worrying is that the ISTQB syllabus builds on old folklore about what testing really is. my worry is not that it is shallow on some parts but that it is plain wrong!

Black-box – meaning that you do not refer to the internal structure – on what level? I find it extremely useful to look at all kinds of structures on all levels I can possibly understand in order to do good testing. I agree with you – it is a bad term that may confuse testers rahter than support them.

Test design: as you say it is the actual analysis and design part that is interesting. The result as descibed here is ONE way of documenting tests and it has nothing to do with the actual design of them.

Then looking at how the basic techniques are described and taught – I have taken the class and even been an (unlicensed) teacher for it a couple of times in the dark ages. The course planning and material is heavily controlled – very little time is allowed for any deeper analysis. If you want to have certified material to use – it must follow strict guidelines. The focus is on teching processes – that are more limiting than helping in my opinion, teaching how to answer muliple choice questions the way the ISTQB or local committe has decided they should be answered.

The whole point is to agree on processes and definitions that are in some cases not very good and other times plainly wrong with your own ideas. It really helps to know as little as possible about real testing – especially about context driven texting – if you want to pass the exam. The idea is to avoid argumentation, limit creativity and produce lots of documentation of limited value.

The whole idea of having a general understanding and agreement is fake. I was for a couple of months memeber of the SSTB in Sweden some ten years back and saw how it worked. Some self-appointed experts – mainly from Germany, Netherlands, Sweden(unfortunately) and England built the syllabus on the German standard and the whole focus is on the standards school of testing. Context-driven – no I do not think so. More context imperialists – changing the context to fit the standard.

I don´t hate ISTQB or the people that work there but I must say that they do not represent MY view of testing. I think the standard is holding us back and leading many down the wrong track. Could you imagine a developer standard that in 2011 focused on waterfall, heavy documentaion, JSP-diagrams etc and not embracing some kind of agile thinking? Do you think developers will respect testers that think and work that way?

Just curious Tobbe, did your fire alarm go off when you finished the comment?

You wrote a good summary of the problem: “the standard is holding us back and leading many down the wrong track”
I don’t think I can change anything, and I spend almost all of my time off-ISTQB, but sometimes you have to have a closer look at the content, so more people have a chance of agreeing what is wrong with it.

It is difficult to see the light if the door is locked, and windows were outside the requirements.

I wonder why so many words and energy are spent on the issue

I teach Software Quality Engineering at several academic schools and it
becomes one of the most popular elective courses! Together with other
professionals who are involved in this initiative, we are pushing new standards at university level education (in Israel) regarding software quality in general and software testing in particular as a mandatory part of Software and Industrial Engineering and Information Systems Engineering programs. Our syllabus covers about 95% of ISTQB material, plus other topics not included in ISTQB.
Hence I ask you: Assuming it will assist them in finding their first job, why not allow our students take the certification exams?

Is it not preferable to direct our energy towards promoting practical teaching and research programs that will greatly benefit our profession? I call all of you to get organized and influence academia to recognize our activities and start paying attention to this academically-neglected important field of education and research.

Dani Almog
BGU

Hi Dani

I have no problems with helping students get their first jobs.
But I have a problem if the help consist of “bad theory” that will narrow their thinking so they won’t do a good testing job.

My question to you is:
Do you think the ISTQB theory is good?
Do you think the test design definition quoted in this post is helpful?

I think software testing is extremely important.
I think text and theory is very important, because it guides and inspires the testing in practice.

It is not my biggest objective to make ISTQB and/or academia theory better.
I rather “Follow the Energy” in my own learning journey, and hope some of the findings can be useful for others.
I hope ISTQB people read blogs, and it would be a positive side-effect if the theory gets better in the future.
You could very well start with a more broad and open definition of test design, I think it can give tremendous ripple effects.

Regards,
Rikard