thoughts from the test eye

Introduction
ApTest Manager is test management system for manual testing. It is easy to use and easy to adapt to your own liking. You have the option to have different levels of features depending on what test project you are in. The more advanced contain ability to set test time per test case as well as determining how long a test actually took among other features. The tool has been created by a company that does test consultancy as well as tools to assist their own testing.

Reviews
The tool focuses on manual testing part of testing and has no support for automated test scripting at all, which is good. Still it is possible to import test result into the a test session (the actual test job) which makes it possible to have the automated tests in the system but when they are run they are imported. It might also be possible to script this import, but I’ve not come that far yet.

Database
ApTest uses an embedded database called Berkley DB. There are various drivers to use the database and here is one of them. It should therefore be possible to link the test case database into external analytics tools such as Spotfire DXP.

UI
All UI:s are web-based, quite good, easy to use and manage. Features such as Search/Replace with Regexp, moving folders, copying testcases are easy to use, even though it is web-based.

Configurability
It is possible to link this into CVS, we’ve not done this ourselves.

It is possible to connect to external bug system by adding fields for bug number, actual link to system. It is also possible to supply information from the test case system into the link using variables.

When preparing a test suite you are able to setup the session variables. These
are variables that can be a single-select list, a multi select list, or any other field you wish to create. Example of these could be IE versions, Web Browser, Operating System and so on. In the test case you are then able to use these session variables by adding <% variable name %>. This is extremely powerful because it is possible to add further context to the test case. Since it is possible to use the test case information when reporting a bug it is possible to have a lot of information ready for copy/paste into the bug system directly from the test case.

The test case ID
The test case ID has always been a topic of discussion. In ApTest the ID consists of all folders and sub-folders plus the test case name, such as /UI/Menus/File/open_file. This means that you cannot have duplicate test case
ID:s in one folder, but you are able to copy a created test cases into other folders easily.

Running a test job
Creating a test session (an actual test job) means that you create a test set where you define which session variables are applicable and then save this as a repeatable test set. You will be able to go back and change the test set if new test cases has been added. When the test set is ready you create a test session that might be applicable for a certain build or iteration. You are able to allocate test sessions to resources and let anyone pick any session. The progress of the test session is visible in a management page for the test sessions. Each test session contain a Summary for the session which shows a minor report of progress and result. It is also possible to go into each test case in the session to adjust result, notes, bugs or other information easily.

While running a test session you are able to edit a test case and update the test session directly.

Reports
There are many reports available, but most of all you are able to create your own reports easily. Adding new fields and different setup of their tables is done quickly.

Scalability
It is possible to work many on the same sessions, test cases etc. The system handle this well.

Documentation
There are two manuals; one for users and one for administrators. All of them are quite extensive and cover many aspects of the tool.

Price
The price of ApTest is far below their competitors. You pay 400 dollars per active online user and 100 dollars per year for each license for their support.

Summary
So far I can highly recommend using this tool. It was a long time since I felt that it was fun to use a test case management system, but ApTest does the trick.

Links
ApTest Website

I’ve found a few handy tools for automating tests as well as setting up a test environment in a scripted way very quickly. One is using python as language with pexpect as platform/module. The second one is for testing web pages quickly using Selenium.

Python is a powerful scripting language and in combination with Pexpect it is very powerful in getting many tests quickly. You use pexpect preferably by CLI-like interfaces such as telnet, ftp and so on. It is also very handy to use when setting up test environments and using the pexpect to validate that the configuration took place. You can use regexp when you validate the result of a command, very nice!

Selenium can be run in Mozilla Firefox as a plugin. You do some small tests, preferably a smoke pass just to see that something is up and running. All you do is recorded and can be rerun easily. You can then export the recorded tests to various program languages such as perl, C# or python.

Usually automated testing or test environments takes some time to setup, but if you just want to use it quick and dirty getting some kind of repeatable result fairly quickly, I recommend the two above.

Being creative is hard when your consciousness is turned on.
Why?
Because your consciousness is an alert cognitive state in which you are aware of yourself in your situation. It is a filter, that not only filter out mostly of the incoming information, but also tries to only deal with information that is important in the aspect of fit into the society. I.e., the social life gets prio 1.

So in order to be successfully creative, you need to train on how to step out of your conscience and enter the world of creativity.

I have realized that composing music is one of my actions to enter this world. When I am in it, I am not conscious. Only creating stuff without reflecting upon myself or awareness of how this music would be accepted by others. This might happen afterwards, but not in the middle of a session. The session is purely an creative session.

There are other actions I do that put me in the same state, e.g., crossword puzzling and playing solitaire card games.
I have even realized that in order to have a successful music session, I play solitaire card games just to get in the proper mode! I.e., I have trained myself into step out of consciousness. And since it is my decision to do so, it is a conscious way to step out of my consciousness.

The same goes for testing. The best and most creative test sessions I have had, have been those where I haven’t reflecting upon myself and my situation. All focus is on processing information and finding out new things.
Not sure if I have found any good way of entering this mode though.
Playing solitaire card games at work is not really socially accepted, yet…
Perhaps compose music might be possible to do? People love those kind of creative skills. That might be accepted… 🙂

I just moved my profile to another domain, which meant I had to setup my Vista machine again.
Besides stuff that makes the computer usable (show all files, don’t hide file name extensions etc.), I try to create many possibilities for software to fail.
You know, a good tester is often lucky…

* run as Restricted User
* use Large DPI setting
* use German Regional Settings
* install support for all of the worlds characters
* non-English language in Internet Browsers
* move system and user Temp folder
* activate IE: ‘display a notification about every script error’
* move Task Bar to the left side of the screen
* Windows Classic Theme & Color Scheme
* use Vista User Access Control
* use an HTTP proxy
* use Data Execution Prevention
* install new versions as they come, e.g. latest hotfixes, MDAC etc.
* never install software on default location

And if you have the possibility, run with 2 screens, on a 64-bit system and with any other settings that might affect your software.

This is a good complement to your test machines; image system and BCM (Basic Configuration Matrix)

This is my reply to the thread “Persuading about Exploratory Methods” in software-testing@yahoogroups.com.

This starts out with the problem that it is sometimes hard to persuade a manager about Exploratory Testing, when all that matters to the manager is that tests ought to be documented (in order to know what should be tested, how many test to produce and execute, etc). Or at least, this is often their argument in such discussion.
And it is also often the case that a tester or test lead find it difficult to explain why it would be suitable with exploratory methods; and exploratory testing in particular. Partly because managers don’t know how to test efficiently; and partly because the tester/test lead find it hard to speak at the same level as the manager.

So here is an example of how you can use the provocative-analogy way when discussing this with your manager.

You:
“Dear Boss, I am concerned about the meetings you attend and what you say on those meetings. I am especially concerned about the coverage of the important issues that should not only be brought up, but also be solved in these meetings…
So, here is what I think you should do:
a) Document everything that you should say on the meeting in advance, by first identifying all possible issues that we have, then by writing down exactly what should be said in a specific order.
b) Write down the expected answer or solution to the issue.
c) Remember to ignore anything else that is brought up on the meeting, the only thing that matters is those issues that you have prepared in advance.
d) Take notes during the meeting (but only when your issues of interest are discussed).
e) Finally, write “OK” for each of your issues where the meeting came up with the solution/answer that you had foreseen, and “Not OK” for those issues where the solution/answer disagrees with yours.”

Boss:
“Are you kidding me!?! Do you have any idea of how a meeting really works?
You see, what we come up with on the meetings are through conversation and interaction.
Yes, there are important things that we know of in advance and that needs to be resolved; we put these on the meeting agenda. But there are also things that come up during the meeting that we could not have foreseen, things that come up during interaction between the attendees.
Also, the answers or solutions to the issues are sometimes very hard to predict.”

You:
“Oh, I see… That it something I can somewhat understand and relate to.
As I understand it, this somehow resembles how we do testing:
We have a conversation with the product and interacts with it to get to know more and more. You see, what we come up with in testing are through conversation and interaction. And yes, there are important things that we know of in advance and that needs to be tested; we put these on the testing agenda.
But there are also things that come up during the testing that we could not have foreseen, things that come up during interaction with the product. Also, what the actual result should be is sometimes very hard to predict.”

Boss:
“Oh, I see… That it something I can somewhat understand and relate to. I have not thought of it in that way. I will never ever tell you to do scripted testing again.
Do the brain-stuff! I am your friend!”

Disclaimer: The last comment might not be said, but hopefully you get my point. 🙂

———-

Another less provocative way is to just use the analogy with testing and meetings/discussion/dialog in your efforts to explain why exploratory methods should be used and promoted.

Well, it is always controversial to criticise the making of the Test Plan (http://en.wikipedia.org/wiki/Test_plan ). But here is an attempt that will leave some open questions for further discussions.

According to my experience, a test plan is a mandatory document that test managers and test leads often promote but seldom question.
Sometimes it is promoted and actually needed; sometimes it is promoted but not needed; sometimes it is not promoted and not needed but still produced anyway.
Why is this happening?
Is it because this is something we can do (by can I mean that this is something we done several times and therefore have developed a skill)?

A test plan is usually done in the beginning of the project and is an old remain from the V-model and Waterfall methods, where all activities are specified and planned in advance (hence the word plan).
But if these project models are out of date, why are we still doing the test plan as if nothing has happened?
Some might say that this, nowadays, is a living document and should change. But isn’t it about time that we call this document something else then? Or break it up into separate documents/artefacts/etc?

One of the main reasons of having a test plan, is the ability to communicate all the how’s, why’s, when’s, what’s, who’s, do’s, don’ts, etc, that regards the testing effort during a project. This is great, but since there will be so many things to cover, it is easy to forget some important aspect or info. Especially if this is something that should be produced in the initial project phase.
And also, what if these things change over time (ever been in such situation?). Is it then fair to expect that the test plan always is updated, at any given time?
Is it also fair to expect that all stakeholders should update themselves by reading this document say once a week? (These documents tend to be large after a while).
And is it fair to say that different stakeholders have different interests in the testing effort? Might they have different quality criterion on the info?

I would like to propose that in many (most?) cases it is good enough to have a Test Strategy to deal with what the testing mission is about; a wiki for documenting the latest useful info; the iteration plan or sprint backlog or similar to keep track of current tasks/work. This way we can address the right stakeholders with information served in the way it would be expected.
Is there something important that we miss if we would do it this way instead?

At one time, I worked as a tester at a company that claims to be at TMM-level 4 (perhaps 5).
Two things struck me:
Firstly, the quality of the actual testing performed on this company was not as good as one could expect from a company that have produced software for over 20 years.
Secondly, at the project post mortem, the testers complained about the lack of respect from the rest of the organisation.

One of the issues with TMM is that it says nothing about the quality of the testing.
A maturity level in TMM is looked upon as “a degree of organisational test process quality”. But it does not say anything about the degree of test process quality; or perhaps more important, test quality.
I think this has to do with the fact that many who promote and implement TMM has little knowledge in actual testing. This means that if you are to improve the actual testing, or tester’s ability to adhere to different test processes, you need to have knowledge about testing and people.
Or put in another words: it is easier to promote a single model than to improve testers’ skill.

Another issue with TMM is that when it is implemented, you might think that it is some sort of assurance that proper work is done constantly. This is not correct.
Since it is not saying anything about the quality of the actual work performed, your colleagues will not judge you on the basis of how high you have reached the TMM stair. Since “quality is value to someone”, you will be judged by your actual work – your performance. Therefore you cannot expect to be respected only because your company has reached level 4 on the TMM.
That is, respect is something you have to earn.

——

The maturity levels, according to the TMMi foundation, can be found at:
http://www.tmmifoundation.org/downloads/resources/TestMaturityModel.TMMi.pdf