Automate configuration checks while testing Martin Jansson
I assume you are familiar with the discussion around checks vs testing brought to you by Michael Bolton, which I agree with.
With configuration I mean settings on a unit such as settings for whatever you are testing. This can be configuration heavy devices such as switch, router or similar using SNMP, applications using the registry or applications using databases for setting storage. Those of you who are familiar with these settings would also be familiar with setting it using a CLI.
So what do I mean with… automated configuration checks while testing?
In a configuration heavy environment there are lots of settings that you know about the system that are stored in a configuration. While working with the system the configuration sometimes changes slightly. Some tests might be to perform a certain task and while finished check if some things have changed or not. You are testing in one area, but you are continuously interested if something is changed in specific configurations.
For instance, in an Ethernet Switch you want to test around the changing of the MDIX, Speed of a LAN port. At the same time you are generating traffic through the system. While doing this you wish to monitor that no other settings are changed, so you might wish to check configurations for alarms. There might be thousands of these settings that you know the state of and that should not be changed, or at least you know what states that they could enter.
My idea is that you create automated configuration checks that polls the system for information either continually or when triggered. The checks are context dependent, but it is fairly easy to know what context is valid in each situation when it comes to these settings. If it is too complex you should perhaps not automate it.
As I see it, a check would best be suited as a unit test (or unit check as Michael Bolton calls it). I am fond of using Python in combination with a unit test framework and Pexpect. Pexpect enables you to create wrappers around whatever you are trying to do with the system whether it is using the CLI or doing SNMP. Pexpect then enables you to check the result using regular expressions, thus enables you to build in some context-dependent checks.
This would enable you to create tools for yourself while doing the real testing.