thoughts from the test eye

As the great Buckaroo Bonzai once said: “No matter where you go, there you are.” To me the whole Checking vs. Testing argument is the same as the prior statment. Think about it.

@Jim.
Well, I honestly think that the Testing vs. Checking series is a wonderful contribution to the community and it has been a much needed distinction that has been great in the discussions about unit checking and sapient testing. Still, there can be paradoxes found in all water proof definitions and theorems. In this case it is a matter of looking at exploratory testing and what that really means if we push it to the limit.

BTW: I thought it was a quote from Confucius, which the film character Buckaroo Bonzai so very nicely stole…

Henrik,

Confucius or Buckaroo it doesn’t matter. ET was borrowed from Cem Kaner by James Bach, and of course they knew each other and James ran with it. My disdain for this whole Checking vs. Testing thing is that it is a muddying of the waters which our profession doesn’t need right now. It all boils down to Emperical/Scientific Methodology in its process.

Besides, if you look up “Sapient” (a word that was originally derived in the 1400-1500’s) it means basically “Intelligent” or “Wise”. Why don’t both Bach and Bolten just come out and say “Intelligent Testing” or “Thinking Testing”. This to me is a consultant’s way of coining a new term to propogate their business.

Why do we need arcane fancy terms to describe a straightforward idea? KISS it and be done with it. There is already too much confusion by groups outside of testing, and within it, of what it all really entails. We are as bad as our developer counterparts in having to re-invent the wheel. There are more important things to discuss and debate than this one. After 20+ years in this line of work I am still amazed by it, both by the technology changes and the in-fighting by the experts.

Me… I’m just a well worn veteran of the Software Testing Trenches.

By polishing our terminology when talking about testing I think we will make it clearer instead of mudding. I am still experiencing that many testers over simplify the way they speak about testing and therefore causes confusion towards personel outside testing.

There are so many things that are fuzzy when talking about quality and work around it. If we identify a few set of terms that makes it easier for some of us, I think you should embrace that.

I think Michael Boltons observation is brilliant.

Martin,

There is polishing and then there is just putting another layer of paint on top of another to make it look new. Terminology/vocabulary is one of the biggest problems in our profession. Believe me I’ve seen my fair share of it over the years and all it does is confuse things, especially with new terms and meanings. Clarification and simplification doesn’t seem to be in some peoples mindset.

There may be some oversimplification, but there is a lot of hair splitting going on too as well. Bolton does have some good points, but again this is hair splitting in order to make a delineation that isn’t needed right now in our profession.

And I have embraced some things over the years that were changes to status quo in testing. I’ve followed Kaner and Bach for years, and they have had some great ideas. But to me the ideas of Checking vs. Testing and “Sapient” Testing don’t change anything I do now or have for years. I always use intelligence and wisdom (called experience) in my testing whether it be scripted or experimental (which is what ET pretty much is) in nature.

@Jim:
Even if you don’t change anything, or if some new terminology hasn’t been of any help to you, it has been very useful for many others. Can you see that?
I somehow agree with you that sometimes there is just new wordings that might not be useful, but many times you need to sharpen the terminology and make distinctions in order to get a fruitful discussion. And Sapient Testing is not the same as Intelligent Testing – hence the new wording. Sapient Testing is “any testing that relies on skilled humans”. So for me, and many others, this has been a term that fits in better than the term Manual Testing which has become somewhat dragged through the mud.
And even if Testing vs. Checking hasn’t changed anything for you and others, the mere discussion about it has been revitalizing and creative. And for many people it has meant a lot with the terming as well.
It might be so that you have been in the industry so long that these thoughts aren’t useful, and it might be the same for experienced developers/programmers. But for all of those that started their careers in the late 90’s or beginning of the new millennium (which was what a lot of people in our industry did) it might shed some new light on things.

Moderator steering… 🙂
The side discussion about terminology is most welcome and should continue if needed, but I hope that there any other thoughts regarding the original post?

Yes, perhaps checking can be considered closely related to Test Patterns (or whatever you wish to call it). If you also consider that a check usually is quick while a test is slower. How would this affect test planning?

There is something intriguing about your paradox (ET being most challenging and appropriate for the novice), but I don’t think it’s totally true, since the tester don’t have the knowledge about the details of the application under test.
You’d have to also include the perfect test cases to reach a checking situation.
The Exploratrory Tester Gods (you know who you are) still needs to investigate and evaluate the product’s behavior, and this is testing.
This leads us to the semantic paradox:
with test scripts you do checking,
with check lists you do testing

Henrik,

Let’s follow the chain here for a minute (please excuse me on this folks).

‘Sapient’ definition according to Websters dictionary online: merriam-webster.com/dictionary/sapient

‘Sage’ definition (link from Sapient): merriam-webster.com/dictionary/sage Especially item 1a.

‘Wise’ (a synonym to sapient) definition (especially used as an adjective): merriam-webster.com/dictionary/wise Especially item 1a and 2a.

Now look at ‘Intelligent’; merriam-webster.com/dictionary/intelligent Look at 1b and 2b and specifically Intellect.

‘Intellect’: merriam-webster.com/dictionary/intellect, specifically 1b and 2.

So how does Intellect/Intelligent relate to Sapient, both talk about some level of knowledge and understanding and experience. A person becomes experienced and wise/sage through the use of intelligence and intelligent actions (i.e., learning). Thus they may be considered sapient (learned). So in plain english lets just say the guy is smart and uses his brain to intelligently approach a problem or situation in order to determine what to do next. Sound like Exploratory Testing?

And I like Rikard’s last 2 lines. Now that is an oxymoron in its best form. This is confusion in the making. Again, let’s be smart about this and not do this type of mishmashing of terminology and understanding so as to come up with something new in order to sell your ideas and consulting services.

I’m a big believer in the KiSS method, Keep It Simple & Straightforward.

And now back on topic to your original post. If you were really that knowledgable and experienced you would be in effect “checking” (validating) your ideas and experience. But you would do so under the premise of proving, or disproving, your ideas/theories via experiments. This all relates back to Emperical/Scientific Methodology and experimentation. And what do we call an experiment in short form in the english language, a Test.

@Henrik Emilsson

>> And there is nothing more to learn since you already know everything.

This “everything” will never be a finite knowledge due to evolving nature and complexity of technical systems including computer programs. Furthermore, some of old “valid” expectations evolutionary become “invalid”, so even the acquired knowledge has to be iteratively revisited.

>> This would mean that all tests that you do are merely checks
>> to see if they meet the knowledge (assumption) or not.

This point is mostly valid when looking from the inside of software development process that aims software product delivery.
But test consultants may see (depending on agreements they have with a Client) testing as a completely embedded process not linked to software development. In this case testing becomes a product exploration process that aims gathering and delivering of information (about product under test) to the Client. In this case you don’t have to confirm, you only need to report; you don’t even need any requirements.

Thank you,
Albert Gareev

@Martin:
Not sure if I understood you completely, but your last remark would be the case when you work with very experienced testers and a well known product and perhaps there is less new functionality. I.e. it should be easy to plan for those tests.

@Rikard:
Since this is a thought experiment, there is room for extraordinary circumstances! 🙂
The thing was that the “great tester” would know about product details. I.e., the more you test, the more you learn, the more you know, the less testing you do (that was the point with the paradox). So the paradox cannot be totally true, I agree, but the point is that an experienced tester will do more checking than testing after a while.
Compare this to when you work with the software you test today, if the only new functionality is that there now is support for a new file format with a new delimiter e.g. pipe “|”, your testing would consist of a lot of checking according to what you already know about similar file formats (CSV). But lets say that another experienced tester that doesn’t know much about the file format CSV is assigned to test the functionality, the testing that this person would do would be similar to what you did when you tested and learned about CSV. I.e., it would be more testing for this person, and less testing for you. Don’t you think?

@Jim:
I leave it up to James Bach for an explanation of Sapient Testing and Sapient Process: http://www.satisfice.com/blog/archives/99
I think that you are right that it could have been called something else like you suggest, but to me (that aren’t English speaking) it sounds good with Sapient Testing – a kind of new wording that hasn’t yet become worn out; and is not easy to mixup with something else.
But it isn’t synonym with exploratory testing! Sapient Testing is any testing activity that requires sapience or intelligence in order to do. It might be risk based testing, decide which bugs that matters, boundary value analysis, or what have we. Exploratory Testing is Sapient Testing, but not vice versa.

Regarding your comments on the post:
I hear you, and it sounds reasonable. But how much experimentation do you do when you test something that you really know very much about? At some point it might become less experimental and more checking in order to confirm some existing belief?
Don’t take this wrong, but to me it sounds very bombastic to do experiments when it could just easily been checked. It is during the analysis of the outcome that you might go back to experimentation if it were so that the confirmation of your belief failed and was something really unexpected.

@Rikard:
Regarding your semantic paradox:
“with test scripts you do checking,
with check lists you do testing”

Well, if we in this discussion talk about Testing vs. Checking, wouldn’t then the “test scripts” be “check scripts”? And the difference between the two is that a script tells someone (exactly) what to do, step by step; but a check list is a list of test ideas up for challenge.

@Albert:
See my answers above regarding the thought experiment and that it is an intellectual experiment.

Regarding your last point:
I get your thoughts. But would there be a difference between an experienced tester and a novice? Would there be a difference in the amount of testing vs. checking done by these two?

@Henrik

>> But would there be a difference between an experienced tester and a novice? >> Would there be a difference in the amount of testing vs. checking
>> done by these two?

It all depends whether you want to confirm it’s right (a single scenario) or you want to find out where it’s wrong (millions of possible scenarios).
Assuming confirmation falls under checking a novice would do checking most of the time. He simply wouldn’t come up with that many test ideas as experienced tester.
An experienced tester knows much more about software weaknesses and misbehavings, he developed certain patterns, tricks, and heuristics for triggering and detection of possible problems, so after checking he could step out in many possible directions – and do testing [endlessly].

So, to answer your question, in that situation amount of checking would be approximately equal between novice and expert. Regarding amount of testing, it depends how inquisitive a novice is, and how eager to learn and practice he is. But for (bug count / time spent) ratio I would definitely bet on expert 10 : 1.

Thank you,
Albert Gareev

@Henrik

Thank you for the kind words; I appreciate the appreciation.

“…the more you test, the more you know, the more heuristics you will develop; the more you become a checker for things that you assume or already know.”

I don’t agree with this, alas. Heuristics are applied, not followed, and they always hold the possibility of failure. Testing, as I see it, always requires cognitive engagement, and so never turns into checking. Testing always dominates checking; checking is always surrounding by testing. To develop checks for any new product, you need cycles of exploration, design, and analysis, all of which are testing activities.

A blog is a place to explore and refine ideas. More time has meant more exploration, so there’s quite a body of writing now on this subject: http://www.developsense.com/blog/category/testing-vs-checking/ In particular, I’d recommend the most recent post in the category (http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/). Maybe those who object to the ideas in the original post will find them expressed more clearly in later posts. I do.

Lets say … you have experience from all kinds of software development and teams/companies. Then you would have so much knowledge about products, people, software, user interaction, etc, that you eventually would sit on a gold mine of experience and knowledge. And there is nothing more to learn since you already know everything. This would mean that all tests that you do are merely checks to see if they meet the knowledge (assumption) or not.

Knowledge and experience are helpful, but the more I learn, the more I’m aware that existing beliefs–both my own and others–are provisional and incomplete. The more I learn about products, people software, user interaction etc., the more variation and complexity I see. Since there are many more people than I in the world, their variability naturally increases far faster than my comprehension. This reminds me (when I’m on my game) to question myself when I think I have all the answers (that is, when I’m not on my game). To me, an excellent tester retains uncertainty when all around are losing theirs.

@Rikard

with test scripts you do checking,
with check lists you do testing

I realized that a while ago (I forget whether I realized it on my own, or if you pointed it out to me first). Some may be concerned that testers might find that confusing. Oh well. If a tester finds that confusing, imagine the problem he’ll have when he realizes that in floating point representation, the decimal stays in the same place, where in fixed point representation it moves around.

I have a lot of faith in testers. I think the best ones can tolerate and embrace plenty of inconsistency, irony, paradox, and humour (as your observation just above demonstrates perfectly).

@Henrik

If someone seriously suggested changing “checklists” to “testlists”, or “test scripts” to “check scripts”, I’d get almost as upset as Jim seems to be. 🙂

@Jim

I’m curious: if you don’t think this is a fit subject for conversation, why keep talking about it?

This to me is a consultant’s way of coining a new term to propogate their business.

If you know a way of coining a new term that automatically generates business, I’d appreciate hearing about it. I’m not aware of any client who has hired me solely on that basis, since I give the the ideas away for free anyway. If clients were to hire me for coining new words or recasting old ones, I suspect that they’d require the new terms to be useful or helpful rather than simply novel. Still, I’m frantically rearranging the dictionary in my Moleskine just in case you’re right.

Why do we need arcane fancy terms to describe a straightforward idea?

I use testing and checking (which are neither arcane nor fancy, in my view) to describe two straightforward ideas. The goal is to attack assimilation bias (arcane, fancy term), or lumping (ordinary, plain term) two ideas that, to me, can and should be helpfully separated. Some people seem to agree; others don’t. If you disagree, that’s okay; you don’t have to use the terms. I’m sorry if they irritate you.

There is already too much confusion by groups outside of testing, and within it, of what it all really entails.

I agree. I’m trying to help clear that up. For an example of people inside testing who have found the distinction helpful in explaining testing (and checking) to those outside, please see http://www.developsense.com/blog/2009/11/testing-checking-and-convincing-boss-to/

If you’d use your last name or some other identifier, perhaps we could find the places where you talk about issues that are important to you, and join the conversations there.

Cheers,

—Michael B.

I coined the term “sapient testing” and I blogged about how I defined it. Anyone who wants to argue based on facts can go check that in my blog.

Words get into the dictionary based on how they are used. We English speakers– all of us and any of us– are allowed coin new words and usages as we see fit. Do you think only the “ancient ones” are allowed to innovate and experiment with language?

The definition of sapient in the dictionary shows clearly that it’s the right word to use as a root for “sapient testing.”

I defined a “sapient process” as one that requires a skilled human pilot in order to succeed. I think that definition is reasonably consistent with the dictionary, while also extending the meaning in a useful way in the context of processes. No one was talking about “sapient process” in any other way before I began to speak this way, so it doesn’t interfere any other common expression.

Sapient testing implies testing that cannot be automated. Checking can be automated. That’s a useful distinction. That’s a distinction I often need to make.

Anyway, I don’t agree that exploratory testing (a sapient process) necessarily leads to checking, any more than driving a route one time means you can let the car drive you the second time while the driver naps.

Thanks for all interesting and thoughtful comments!
I now realize that the paradox isn’t true, and cannot ever be. It was an intriguing thought that ended up being something not really true. Thanks for your help in clarifying that!
Still, this post and comments have at least made me think around the subject! And it has made me wiser.

I’ve found this testing vs. checking discussion to be very interesting, and therefore did a lot of blog research on the debate. I wrote an article for the June 2011 issue of the Automated Software Testing Magazine where I summarize my research. Basically, it seems that most people agree that testing is composed of verification and exploration skills, and don’t find this to be a new concept. Most of the disagreement is with the redefinition of the word test. I quoted passages from this blog and comments from this blog in the article that I wrote. Find it at http://www.astmagazine.automatedtestinginstitute.com