thoughts from the test eye

Not too long ago I had a test lead that I pretty soon recognised as someone that didn’t share my philosophy in software testing.

One day I reported to him that I had run all regression tests that were assigned to me.

All tests were executed on the same build (we had monthly iterations and this was done in the first week); and a couple of bugs were found and reported. I also verified a couple of bugs that I had reported earlier and that had been fixed and included in the current build. The suite took 3 days to execute and it consisted of rigorously documented tests that had little to offer to your fantasy. The system that was tested did not have any UI and the final results could only be checked via a secondary tool and/or by monitoring several log files; so it was a bit cumbersome to test.

I guess that I was pretty fast or efficient because the suite was supposed to take 5 days or so to run, so I was rather glad to have done this in almost half the time; and believe me when I say that I wasn’t sloppy.
And especially if it is a rather boring task I tend to think that it is better to do it as quick as possible, without being sloppy, in order to get rid of the boring stuff.

Well, after I had said that I was finished I expected to hear from my test lead what to do next.
Perhaps having a look at the new functionality that we should have tested last iteration but didn’t have time to do (i.e. we were a bit behind the schedule)?
Or perhaps helping the other testers out with their suite?
Maybe verify some bug fixes reported by others that were included in the build?

No, this was nothing that he said. Instead his response was:
“Hmm, OK… Could you test it again?” And I saw that he was trying me with this sentence because I caught a nervous smile in his face that he was trying to hide.

I just shook my head and said “Are you serious? What sense would it make to run it twice?”

He did not have any good answer to that. And of course I did not run the same tests once again…

But that incident made me wonder how many times he had done that to other people before. And in general, I wondered how many test suites are rerun several times on exactly the same configuration just in order to make it look good for some reason.

Sometimes it is said that each test case must have an expected result, or even worse, that each step of a test case must have an expected result.
This is the extreme of scripted testing that I dislike for two reasons:
* It takes a lot of time to write and follow detailed test cases; time that is better spent testing things you haven’t had the time or enough information to thoroughly document.
* It takes away the freedom of the tester to try things a bit different, so we will learn less, narrow our thinking, have less chance of finding new, interesting information, and it will be boring to test.

I understand that the reason for the details is that you want to be sure exactly what has been tested.
But the problem with this reasoning is that in software testing, there are so many things that aren’t tested. You are not testing on the exact same data that your customers use, you are not testing on the exact same hardware and interacting software, and you are not testing in the exact same way as the customers will use your software.
So we can be sure which tests have been run, but that doesn’t necesarily mean it will work for the customers.

This is an example where (Bad) Process is way too much more important than Content.

I prefer a lot of one-liner test ideas, that either can be used in vague test cases or as starting point for Exploratory testing.

A few years ago I switched company from a software company to a hardware focused company. I understood from start that this would be a difficult task in getting to learn the new technologies and platforms, but I did not see what this would do for testing.

When you test software that you are totally new, but where you have experience with the environment such as operating system as well as technologies used, you are still able to dig deep into testing and will probably find lots of bugs.

If you instead test something where you are both new to the software/hardware/system and to its environment you will have it a lot harder to find bugs. You will stumble upon things like… oh this went wrong, then learning ah I did not configure it correctly or oh it is supposed to work like that. You are unsure what to expect and what is actually wrong.

Think of yourself as an explorer where you have no knowledge about the gear you are using, where you have landed, what this green thing you stand is and so on. Do you dare explore? That hot red thing over there… let me just dip a toe. Oh it was lava.

So, when you find yourself in such a situation you should do some research, read lots of books and get to know what territory you are in. Having knowledge about the environment and platform you stand on will enable you to do a better job as a tester.

”Trac is an enhanced wiki and issue tracking system for software development projects. Trac uses a minimalistic approach to web-based software project management. Our mission is to help developers write great software while staying out of the way. Trac should impose as little as possible on a team’s established development process and policies.”, see http://trac.edgewall.org/

This is a very nice light-weight tool. Some of the features:

• Got a wiki
• Possible to connect to SVN to see changes
• Easy to link check-ins with tasks, bugs, etc
• Setup milestones for planning
• See timeline for project management

I can highly recommend this tool if you want to do project management to some level while at the same time utilizing the benefits from SVN.

From what I have seen there are few recruiters that know what testers and test leads do. Considering that we had a EuroSTAR focusing on defining the profession can be seen as that testing as a profession is quite new and can therefore be considered immature.

Many recruiters look for a certain set of skills, education, certificates and knowledge in tools when seeking testers and test leads. The emphasis on tools and certificates seem to be great. They are unsure what keywords to use when searching for the aces in testing.

There are so many parameters to think about when recruiting testers and test leads, which makes this a hard task. Consider that each company is different, each tester might work differently, each object that needs testing is different, and there are a multitude of technology areas that need different skills and so on.

We assume that persons with different background, gender, education, skill sets and so on will generate different ideas on how to test and will probably come to different results. Transferring this fact to recruiting will make it hard for a recruiter.

I have no magic formula for what key words to look for when seeking testers other than that they should be good at testing, which I’d figure is hard to filter out in a job application. Something not directly related to recruiting tester but you may find helpful is SmartRecruiters’s white paper on CRM’s.

Cem Kaner has written a good article about things to consider when recruiting. This is not only for test related jobs, but for recruiting as a whole, which you can find here.

Johanna Rothman has also written a good article on recruiting testers, which you can find here.

This was originally a response to Rikard’s post “Multi-Dimensional Software Testing”, but here I have developed my thoughts a bit.

As I see it, there are more or less obvious stakeholders and stakeholders that might be more or less hidden. A “customer” might be such an obvious stakeholder. It might then just be a matter of recognizing which of these “customers” that matter and take those expectations into consideration when selecting test activities.

The project manager is another example of an obvious stakeholder.

And a product owner would naturally be considered an obvious stakeholder.

But how about the hidden stakeholders; and what are their interests in the project/team?

Maybe the members of the test team would have interests in the team’s work that would affect the testing activities (i.e. their own work)? Someone in the team might be dependent on other peoples work. Some people in the team might lack important skills which would affect the chosen test activities. Are there some people that only can work part time and thereby affect which times meetings are held in the team?

The members of the test team are them self stakeholders to the project and might have important interests. This might have impact on the testability, documentation effort, release schedules, communication, etc. Project members other than the testers are also stakeholders to the project and they might have their specific interests.

Business managers might have much to say about the testing activities; and they might especially be interested in the cost of the testing activities. They might have influence on hardware or software that would be needed and thereby affect the testing activities.

Other test groups that work in parallel might be interested in the outcome; and especially to make sure that your team doesn’t over-deliver and make their team look bad. And they might be interested in what type of cooperation that could take place in order to be more effective. They might also have an interest in what type of reporting to do in order to compile a common test report. Other test groups might also be interested in creating a common vocabulary to use in the test scripts.

The maintenance team that would receive the product after it has been released might be interested in the outcome. And especially they might be interested in the format of the test product i.e. documentation, test scripts, procedures, etc. They might also be interested in good practises that have come up during the project; and if these good practises were designed to work for the maintenance team, they have indeed already affected the testing activities in your team.

Certainly there are more hidden stakeholders than those above to take into consideration when selecting test activities or test strategy to focus on. But my goal is to illustrate the problem with a couple of hidden stakeholders in order to make you think about your situation and possible stakeholders that might matter for you.

Note:
Many of these stakeholders are already taken into consideration on a daily basis, but I believe that they are not considered as stakeholders. Maybe you would discover more important aspects of each stakeholder if you tried to analyse each of them more thoroughly.

E.g., try to list all people or roles that matter to the team and start to analyse what their interests are and how they might affect your team/project.

And you should not be surprised that some of these were hidden to you at first, and when their interests are revealed you might discover that they would have a large impact on your testing activities.

The indispensible worker is the one who always saves the day by touching the program by giving some magic input in order for it to work. He will quickly get the work done and make the customer happy. Upper management will see him as the perfect employee, but many co-workers will think differently.

In the indispensible workers back-water you will notice half-finished tasks, programs that need the magic input in order to work, documentation that usually are not the latest version or just not there, hidden tasks that was performed once a week manually to avoid the weekly disaster and so on.

In my experience, if someone has become indispensible, where the company cannot work without them being there, it will often lead to disaster. This can easily be seen if someone is sick, on vacation or if they leave the company, when certain things start to break down or that the team stands still waiting for the magic touch.

As a co-worker you avoid these traps by requiring documentation enough for someone else to perform the task or that you have at least a backup for the critical tasks. The hardest thing is getting upper management to see that it is not often good to focus on the short term solutions where you save a small bit of time in the moment, but where you have to pay for it tenfold in the future.

Thare are many ways to look at the problem of testing software, and it is rarely wise to use only one; that’s why there are so many mnemonics, e.g. SFDPOT, CRUSSPIC STMPL, HICCUPPS, FCC CUTS VIDS (http://www.satisfice.com/tools/htsm.pdf, http://www.developsense.com/articles/2005-01-TestingWithoutAMap.pdf, http://www.testingreflections.com/node/view/2823)
Here are some questions for your confusion:

1) What?
a) The functionality, what the software does from A to Z; what elements the software works with; the source code, from beginning to end; the requirements, the use cases, the test cases; the product as a whole, including documentation and support. None of these give the full picture.
b) The actual tests, their artifacts, the communication.

2) How?
a) The attributes of the functionality; Usability, Performance, Security, Reliability, Installability, Compatibility, Scalability, Accessibility.
b) The process (both the documented and the real) used when testing; your techniques and approaches, your tools.

3) Who?
a) The (potential) users, which you probably can’t bring down to a couple of personas, but it is good to know what the users are like; what language they speak; what most of them knows.
b) The testers, your (different) skills. Will the developers also do testing? Will customers test the software, in order to accept the product or find bugs?

4) Why?
a) What do you think the users are trying to accomplish with the software? Can the software be used in many more number of ways as there are users? Are there any unintentional effects that are desirable? Why should the user not switch to your competititor?
b) What are the main drivers for the testing effort: Find all relevant bugs; provide enough information for the project to succeed; minimize customer complaints; learn and explore?

5) Where?
a) The environments where the software will operate and interact. Specific hardware and software?
b) Which environments can you afford to use; how far can you get with Basic Configuration Matrix (http://thetesteye.com/blog/2008/05/bcm-basic-configuration-matrix/)

6) When?
a) Will the software only be used at one given time, at one place? Will upcoming releases replace the old version, or is it necessary to be forward compatible with currently non-existant software?
b) When will you test the software, and in which shape will it be?

At my former company we had a vision of removing/fixing all unhandled exceptions that we found (and that we thought a customer might encounter). After some discussions we decided that the unhandled exceptions should be treated in the same way as we had treated severe errors on previous platforms. I.e. Crashes in C++; NullPointerExceptions in Java; “Runtime Errors” in VB; and “Script Errors” in JavaScript.
This was a philosophy that both testers and programmers agreed upon (more or less) and it fitted our context.

Now, for the last two years I have been working as a test consultant; and during these years I have been on a couple of customer sites where their applications have been developed in C# .Net.

When unhandled exceptions occur, many team-members and users that I have been observing just click “Continue” and continue with their work as if nothing happened. And when I afterwards have asked them why they did so, it appears that they never reflected upon the error and if it might have been severe.
Many say that they don’t see the unhandled exception as a crash, or “Soft Crash”, instead they have seen it as a warning or an error that “really shouldn’t be in there”; some people see it as an error that in fact is handled, hence the error message and the pop-up dialog box.
Sometimes I point out that after an unhandled exception you cannot be sure about the state that the application is in, and that any behavior afterwards might be affected by the first “crash”. But I rarely hear anyone else saying this.

I am not saying that hard crashes are to be preferred in any way, but then you at least understood that something was severe. 🙂

I think that soft crashes give users/developers/testers/stakeholders/etc a false sense of safety.
And I blame the “good handling of unhandled exceptions” that the .Net framework provides. In fact, I blame the “Continue” button of those dialogs.

What are your experiences and thoughts regarding this?

(This was originally posted as a new thread on the Context-Driven Software Testing list:
http://groups.yahoo.com/group/software-testing/message/5794 )

I’ve recently become addicted to baking bread. I don’t know why, but it has the same kind of magic as music and software testing; so I’ll try to make some analogies.

Results – The best bread results come when there is long fermentation time, just as when a tester can spend a long time with a product.

Content – There is no need for many ingredients to get an awesome result, but they need to be of high quality, and be handled with care (which also includes banging the dough really hard, as when proviking a crash.)

Recipes – You can follow a recipe (or a test case) and get a good result, but it will be even better if you observe the object and make modifications as the project evolves.

Best Practices – There are a lot of best practices for bread baking, and some of them are totally opposite. They are great to have as guidelines, but different bread requires different practices. You will also get your own best practices, just because that’s the best way to handle dough in your kitchen, in your project.

Tools – A way to bake the dough is needed, but apart from that your hands and mind are the only necessary tools. On the other hand, a kneading machine makes it easier, and a lame bread slasher can be handy. And you have to know when to use the tool, and when you shouldn’t.

Automation – In the shops I can buy bread made without human interference, they are edible, but don’t taste good, and are not healthy. A person that at least overlooks the process and make necessary alterations is needed for a really good bread.

People – Bread (and software) is about people; people that make’em and use them. A perfectly made bread is of no use if noone wants to eat it.

Context – Some bread are perfect for breakfast, and others taste best together with dinner, and some bread might only work with a special kind of soup. Some bread are your favorites.

Sometimes I joke and say that a tester is being really lucky when he discovers a very important defect.
But it’s just like the baker; it isn’t like he is opening the oven and saying: “look what a wonderful bread I found!”

And for your own baking: buy a proofing basket; use any recipe, but add more water and halve the yeast and double the fermentation time; feel the dough and smell the bread in the oven.