thoughts from the test eye

I’ve recently finished James Bach’s book Secrets of a Buccaneer Scholar. I liked it, but I don’t agree with all of it. As a tester, I feel that it inspires me and gives me new ideas in my way of thinking and how I perceive learning, especially self-education. I fully agree with James on that you should follow your passion. If you are able to assist those around you to that end, it will make you grow even more. In march James will be in Sweden and hold a series of courses, one of them is Self-Education for testers. It think it would be very fun and educational to attend, I will see if I can make time.

One thing that I consider after reading the book is how I can inspire my daughter to learn, test new things and to follow her passion. When she receives a new toy, I want her to explore how it works. For instance, she got a little toy puppy that execute somersaults. It had a lot of mechanic inside it and sounded like it was very fragile, if you actually played with it. The purpose of the toy was probably just to watch it. I dislike such toys. I asked my daughter how she thought it worked. She did a somewhat exploratory, destructive test by enabling the puppy to do the somersault, then directly afterwards hugged it tightly. There was a popping sound in the mechanic as the puppy tried to execute the somersault, while being held secure. After that test it was not possible for the puppy to somersault, rather it performed a half one and landed on its nose. I applauded my daughters destructive sense of testing of the toy. Translating that into testing terms, she prioritized which test to start with and considered what was the biggest chance a user would do then executed that. One test to render the system useless. Wonderful!

I think we have a lot to learn (or perhaps relearn) from our children.

Now and then I read that testers don’t know enough about Usability, that there is a need for a Performance Testing expert, that a Security consultant should be called in, or that a master of the used technology would make Installation and Compatibility testing possible.
This might be true in the general case, but there are many testers that are working with the same product suite for years, and over time you become an expert of most aspects of quality for your specific product.

Let’s look at some examples:

One sub-category of Usability is Operability; that an experienced user can perform functions in a fast and efficient way. As a system tester, you have done most operations many times, and you know what you can expect from features in the sense of operability. You can point at places where for instance the ability to delete multiple items with a few clicks would make a difference.
Learnability is another sub-category of Usability. You only learn the basics once, but maybe you have heard customer stories of confusing things, or you could let a new member of the test team think in this direction.
Regarding Accessibility it doesn’t cost too much to use High DPI, speakers turned on, Code Blind or Magnifier sometimes.

Large-scale Performance might require an expensive tool, but I bet you’re doing some simulations without them; maybe just by telling the whole development team to hit the same server at the same time.
There is also a small-scale, low-level Performance aspect that shouldn’t be understimated. If a dialog takes more than a second to display, it might be something that put the user’s confidence in doubt.
If you have tested your product for quite some time, you will immediately notice when something takes just a bit longer than it could take. There of course might be valid reasons for this, but talking to the developer about it might be beneficial to both the producers and consumers of your software.

Security testing seems more difficult, but as a product tester you know at which moments authentication takes place, you know if there are passwords stored, and that they should be encryted; you might not know how to exploit a crash, but you are an expert at provoking the crashes.

For Hardware/OS/Application Compatibility testing you will become more of an expert the longer you work with them, at least if you have the curiousity to learn more things when you get the chance.
You might not know a lot about how the iPhone works, but you know all the details that can be used for interacting with your web site.

Sometimes I also read the extreme that testers shouldn’t bother with Usability/Security/Performance testing, which to me seems like an incredible waste of knowledge and resources.
When testing functionality manually, you can look at quality attributes at the same time, and get a lot of coverage for free.

I’m not saying true specialists aren’t needed, but I’m saying that there are a lot of expertise in your building that at least can be used complementary.
If you are in a situation where you know a lot of these things, but aren’t allowed/encouraged to test these things, I think you should try to convince your managers of a better and more fun way to test your product.

I have been thinking about qualitative research lately, and wondered what the question(s) would look like if testing was seen as a research project.

The testing effort has many positive effects, but one common and important is to provide information about the product, so a good release decision can be made. We cannot prove that the software is good, but we can try our very best to show important ways it can fail.
But at the same time, it is too late to come with important information just before the release is to be made; the information should be provided in a timely manner.
So in a paradoxal way, we try to destroy the product we love, but at the same time give the information to the project, so they can make the testing mission fail.

Testing try to answer “No” to the “Can we ship?” question – many times in many ways – but gives the project the best chances to resolve all issues as early as possible.

Another important question is “How can the product be better?”, and information about this is also being provided all the time. Not too late, but also not too early, in times when we know too little, and the cost is too high to reach the vital information.

Of course, there are many other sources and considerations, but testing often has the best view of the system as a whole, and the best view of some details from a customer perspective.
Testing can’t be complete, but the answers are valid when the testing can be considered saturated, when too much effort is needed to get new information that aren’t too important.

Are there other questions that are more important?
Or are we providing answers without questions?

Cem Kaner has updated his set of publications. I’ve been reading his well written articles over the last ten years.

Have a nice time digging in!

Jerry Weinberg has, in his book “Perfect Software and other illusions about testing”, expressed a very important observation, namely who is responsible for pinpointing the bug. The tester finds the bug, tries to reproduce it, then adds as much information that he/she has such as log files, configurations, test data and so on. When you estimate time for testing I think you most often consider the time that you do to find the bug and then make a report. As Jerry points out, the time it takes to pinpoint the exact location is not taken into consideration. It might also be unclear who has this responsibility.

This might not be an issue where there is high testability, when it takes little time to report the bug and when the actual pinpointing takes little time. When testing complex systems (a system consisting of several sub system, different hardware, many interfaces, a vast amount of ways to get information on what actually went wrong), the time for pinpointing can be a factor that is underestimated and that can really be a major issue in a project. Reproducing a bug can take several hours or possibly days. If you, after having reported the bug, are asked by the developer to add extra or missing information to the bug it might cost you days to get into the same erroneous state. In some cases it might be close to impossible because of missing equipment or missing/broken units. When it takes this long time it is inevitable that a conflict grows in the project. It is also inevitable that the testers start to doubt how much time they should spend on bug reporting (in the worst case scenario consider not doing it) and if they can help developers.

How do you handle pinpointing? Is there any need to increase testability to make it easier to report bugs and get information from the sysem? How have you discussed and communicated this within our projects? Is your testlead and project manager aware of this time sink and uncertainty?

If you want to taste some of Jerry Weinbergs knowledge I recommend reading this book.

A disclaimer… I am no developer, but I have developed a tool. As I develop I have the mindset of a developer, not the tester. I have done lots of mistakes, intentionally not implemented good/needed things and considered what parts I can get away with in the first release. This tool might not seem big and useful, but I have used it and it has created many interesting results in the past. As I developed this I tried a new method of implementation… all ideas I had on what functions the tool should have, what was supposed to work, what was not supposed to work etc I wrote down in a testideas-document. I then had one column that identified if it worked or not in a specific release. All good feedback I added to that list.

This is the first tool we create at the test eye that is open for the public. At thetesteye we have choosen to publish our material under the license Attribution No Derivatives. My personal aim with this tool was to increase my knowledge of coding. I have used Python and Tkinter as a graphical interface. In the Publications section you will find the link to the tool and the currently released version.

General discussion

The general idea is to use the frequency of words as a way to find errors. The more text you analyze, the higher statistical significance; thus resulting in an easier chance of spotting the erroneous words. This kind of script is very often found as a code example. When I first created a script for this I did not know that. I ran it on a quite large text corpus and found that the company name had been spelled incorrectly 7 times in the copyright text. I also found lots and lots of spelling mistakes as well as some strange API functions that were incorrect.

Use cases

• Run on documentation to find unfrequent words (that usually contains spelling errors)
• Run on code to find variables that are similar but not the same and used incorrectly
• Run on code to find unused variables, thus variables only used once
• Run on code + API documentation to find things that should not be there or code that are not covered anywhere
• Localization specific: When doing translations you might be allowed to have a certain amount of errors, this is one way of finding a few extra faults that you can remove

How I use it

I run the tool on a tree structure. I open the result file in Excel or OpenOffice Calc. I then sort on frequency… start deleting uninteresting records. You can open it in MS Word or something similar to filter out things that are in fact spelled correctly. After a few cleaning ups you might have a list that is worth investigating.

Bugs and Enhancements

The testideas.xls contain the current tests and some of the enhancements that I’ve gotten so far. If you got any suggestions, feel free to mail me at martin.jansson@thetesteye.com.

Let me begin by saying that these are my beliefs ever since I took the ISEB/ISTQB certification. But when I thought of this recently, I think I need to make a statement and try to help all those that are rejected because they are not certified.

————————————–

In the search for a qualifying certification many European companies have chosen to use the ISTQB certification and specifically the Foundation Level as a qualifier. But there are several voices raised against the validity of this certification, many prominent experts do not agree with the content of the syllabus. The problem gets worse because the more people that gets certified, the more this certification becomes “valid”.

The certification is based on an examination that you can take after reading a syllabus by yourself or paying for attending a two-day course and then take the examination. Almost anyone can do this! And it does not say anything about your testing skills. So this certification should not (and cannot) be used as a qualifier of how good one person is as a tester.

So what it really means is that a person that hold this certification has managed to get at least 30 multiple-choice questions right out of 40.
Think about that before you reject job applicants because they aren’t certified (which seems to be the case too often in Europe).

Let us treat software testing as a serious profession and throw out this certification as soon as possible.

In many discussions about testing methods, courses, techniques, approaches etc it is usually software testing that is in focus. I cannot see why the limit is set to just software. For instance, the excellent course Rapid Software Testing advocates, by its name, that is meant for personel who perform testing of software. It could perhaps be called Rapid Testing, because the esssence of the course is not limited to just software. Does the name limit what non-software testers and their managers think of this course?

I’ve seen similar limited thinking when someone was building production test equipment to be used on each unit created in the factory. For production testing there are some important factors of the tests performed namely speed of the tests and a high yield. The production test equipment should also be user friendly because the person running the test should do it correctly and quickly so that you get even higher yield.  You can use the concept of a smoke test, then take some ideas on how unit testing is done but there are several other areas that can be considered when creating the tests in the test station. Naturally you can use your entire tool box of techniques when testing the test station/production test equipment.

When testing hardware you do many checks (as Michael Bolton calls them), measuring power, temperature and so on, but you can do these with an exploratory approach. There are usually several well defined tests found in various standards with test scripts on how to execute them. Usually it is the designer who does this task. Do they use any of the methods and techniques that are created and discussed in software testing? I think that would be very rare.

Me and a collegue used exploratory approach and pair-wise testing when doing testing on a radar used to trigger a blinking light. There is some software included in all this, but I would not say I was doing software testing, rather system testing or just testing. I was not doing any verification or validation, since we had no specs and the creators had given us too little information on how the system was supposed to work. So, we explored it and found bugs that did not fit in our model of thinking. Some of the issues found were indeed bugs others were not.

Do you say that you do software testing when testing a car? 20 years ago software was probably never used in a car, but the test ideas and test techniques from that time might apply today with the cars that do have software to control all major systems.

Before being too quick in saying that it is software testing that your approach, technique, course or thingie is focusing on, do think again.

For quite some time I have felt that the classic test design techniques don’t add up to the needs of software testing that tries to find most of the important information.
At EuroSTAR 2009 it dawned on me that it is time to describe the method that I, and many, many others, have been using for a long time.
By talking more around this, I hope we can spread the usage of this test design, improve the way we do it, and make it more understandable for other people, avoiding the comment: “oh, you’re error guessing.”
What I will describe could be called a method, activity or a skill, but I like to think of it as a test design technique, that is seen as a tester’s most powerful weapon, much better than equivalence partitioning, decision trees etc.

Grounded Test Design: learn a lot, and synthesize important test ideas.

To explain this further: Software testers should follow the thoughts of social science Grounded Theory; we should use many sources of information regarding the subject (requirements, specifications, prototypes, code, bugs, error catalogs, support information, customer stories, user expectations, technology, tools, models, systems, quality objectives, quality attributes, testing techniques, test ideas, conversations with people, risks, possibilities); we should group things together, use our creativity, and create a theory consisting of a number of ideas capturing what is important to test.
Grounded Test Design is for those that want to test a lot more than the requirements, that understand the need to combine things of different types, to look at the whole and the details at the same time.
This might seem heavy, but people have phenomenal capacity.
It has a lot more to it than the closest technique error guessing; for instance, it is not only about errors, it can also be investigations, or questions regarding dubious behavior.
It isn’t guessing, it is a qualified synthesis of knowledge.
And also: it isn’t (and doesn’t sound like) something done lightly, something unstructured, or unprofessional.

Being a technique, it is something that can be applied in many different situations. The most informal Grounded Test Design might happen extremely fast in the expert tester’s head when thinking of a completely new idea while performing exploratory testing; or it could be the base for test scripts/cases/scenarios, that are carefully investigated and designed in advance. Good software testing uses many different methods.
A formal Grounded Test Design (to be defined, tried and evaluated…) might be best used by someone completely new to a product, or maybe most efficiently performed in diversified pairs.

Grounded Test Design has a loose scientific, theoretical base in Grounded Theory (Strauss/Corbin), used in social science as a qualitative, thorough analysis of a phenomenon.
The scientists examine the subject very carefully, gather a lot of material, document codes and concepts, combine them to categories, from which a theory is constructed.
There is no hypothesis to start with, as in the traditional natural science, and that’s quite similar to software testing; it would be dangerous if we thought we knew the important questions at once.
I think it is good to be slightly associated with this theory, especially since it emphasizes that the researcher must use her creativity.

On the other hand; do we really need this word, maybe we already have enough words about how to do good software testing?
Or is it a good word to have in reserve when explaining to managers why testing is difficult, takes a lot of time, and must include manual work?
Or should we go further with this and try to develop a framework for performing structured analysis of everything that is important for testing?

Opinions are very welcome.

Software Potato

 The square symbolizes the features and bugs you will find with test cases stemming from requirements (that can’t and shouldn’t be complete.)
The blue area is all bugs, including things that maybe no customers would consider annoying.
The brown area is all important bugs, those bugs you’d want to find and fix.

So how do you go from the requirements to “all important bugs”?
You won’t have time to create test scripts for “everything”.
So maybe you do exploratory testing (thin lines in many directions), and hope for the best.
Or maybe you test around one-liners (thicker horizontal lines), that are more distinct, that are reviewed, and have a better chance of finding what’s important.
Either option, some part luck, and a large portion of hard work is needed.
But I think you have a much better chance if you are using one-liners, especially if it’s a larger project.

Later I have realized that one-liners aren’t essential; that this problem has been solved many times at many places with many different approaches.
What is common could be that testers learn a lot of things from many different sources, combine things, think critically and design tests (in advance or on-the-fly) that will cover the important areas.
Maybe we need a name for this method; it could be Grounded Test Design.