Lightweight Compatibility Testing Rikard Edgren
In testing text books you can read that compatibility testing should be performed after the functionality testing and bug fixing is completed. I guess the reason is that you don’t want to mix these categories, but hey, what a waste of resources.
My suggestion is to perform the compatibility testing at the same time as you are doing your other testing; when problems arise, trust that you will deal with them.
In my classification, compatibility testing involves hardware, operating system, application, configuration, backward/forward compatibility, sustainability and standards conformance.
Here follows some lightweight methods to tackle these areas.
Basic Configuration Matrix
Basic Configuration Matrix is a short list of platform configurations that will spot most of the platform bugs that could exist in your currently supported configuration matrix.
The simplest example is to use one configuration with the oldest supported operating system, oldest browser etc; and one configuration with the newest of all related software. A more advanced example could use several configurations that use different languages, Application Servers, authentication methods et.al.
Often it will take quite some time to run most tests on BCM; so alternate between the configurations while testing your product. Do variations on configurations when appropriate.
Error-prone Machine
Another trick is to setup machines so they have a high chance of stumbling on compatibility issues. You can vary this on your BCM, your personal machine or whatever is suitable. The idea is to get some compatibility testing almost for free.
Examples on Windows include: run as Restricted User, use Large DPI setting, use German Regional Settings, install support for all of the worlds characters, non-English language in Internet Browsers, move system and user Temp folder, activate IE ‘display a notification about every script error’, move Task Bar to the left side of the screen, Windows Classic Theme & Color Scheme, use User Access Control, use an HTTP proxy, use Data Execution Prevention, install new versions as they come, e.g. latest hotfixes, MDAC etc, never install software on default location, run with 2 screens, on a 64-bit system, use different browsers, turn off virtual memory swapping, , install Google/Yahoo toolbar, run Energy Save Program, pull out the network cord every time you leave the computer; and put it back in when you return, turn on the sound!
Technology Knowledge
If you know a lot about the environment the software operates in, you know which things will happen in reality, which settings that usually are altered, and how it is commonly operated.
The lightweight method is to use this knowledge and make sure you test the most important things.
Letting Others Do the Testing
Many compatibility problems happen on basic usage, which indicates that you can let others do a big part of the compatibility testing: developers can use different machines and graphics cards, Beta testing can be done in customers’ production-like environment. If the product is free of charge, you might even get away with addressing problems after your users encounter them (but make sure you have an easy and compelling way for them to do this reporting.)
Crowd-testing could be a way, but so far the payment models from the testers’ perspective are not ethically defensible, to me.
Reference Environments
To quickly investigate if you are experiencing a compatibility issue, it is handy to have reference environments available. It could be someone else’s, a virtual machine, a quickly cloned image, your own machine etc.
Personally I prefer having a physical machine that is running similar things, but on a different OS, different language and/or earlier version. The last years I have had three machines and three monitors, and by switching, I get a lot of compatibility testing done at the same time as testing new features. When I check things on an older version, I can save documents and use them for next tests.
Backward/Forward Compatibility
Backward compatibility is easiest done if you can use real customers most complicated files/data/documents. Use these as you test any functionality (Background Complexity Heuristic).
Occasionally communicate between different versions.
Forward compatibility should be designed in the architecture, as a tester you can point this out.
Sustainability
Have conversations around the question: Is the product compatible with the environment? Have we considered energy efficiency, switch-offs, power-saving modes, support work from home and the likes?
Standards
A lightweight method for standards conformance is to identify which ones are applicable, and ask the experts if they understand it, and successfully have managed to adapt it to the new context.
Let’s finish with a non-lightweight method: you can become the standards expert.
Hi Rikard
Good post! We’ve ended up doing the compatibility testing at the same time as we do other testing as well. The chief argument is that the problems uncovered during compatibility testing are often critical to users with that setup, and that the bugs tend to take a long time to fix, meaning that we need to find them early.
But could you elaborate on this statement please?:
Crowd-testing could be a way, but so far the payment models from the testers’ perspective are not ethically defensible, to me.
With crowd-testing I mean sites that connect many testers with many products.
I have seen three payment models:
* pay per accepted bug
* bug competition with prize to winners
* hourly rate way below a “regular” job
These give situations where many testers are under-paid, or not paid at all (e.g. if you find zero, or duplicate bugs)
I don’t think this is right.
There are understandable reasons for current models:
* difficult to trust testers you don’t know
* middle-hand competes with low cost as selling argument
But I think you either pay people properly when they do work for you, or you don’t do this at all.
I don’t have any solutions, except a generic
more trust in the world
Really good post! Loved it!
sussla.wordpress.com