thoughts from the test eye

As a tester, part of your mission should be to inform your stakeholders about issues that might threaten the value of the system/solution. But what if you as a tester do not know the boundary of the system? What if you base your confidence of the result of your testing on a fraction of what you should be testing? What if you do not know how or when the system/solution is changed? If you lack this kind of control, how can you say that you have confidence in the result of your testing?

These questions are related to testability. If the platform we base our knowledge on is in fluctuation, then how can we know that anything of what we have learnt is correct?

An example. In a project I worked on, the end-to-end solution was extremely big, consisting of many sub systems. The solution was updated by many different actors, some doing it manually and some doing it with continuous deployment. The bigger solution was changed often and in some cases without the awareness of the other organisations. The end-to-end testers sometimes performed a test that took a fair amount of time. Quite often, they started one test and during that time the solution was updated or changed with new components or sub systems. It was difficult to get any kind of determinism in the result of testing. When writing the result of a test, you probably want to state which version of the solution you were using at the time. But how do you refer to the solution and its version in a situation like this?

When you test a system and document the result of your tests you need to be able to refer to that system in one way or another. If the system is changed continuously, you somehow need to know when it is changed, what and where the change is as well. If you do not know what and where there are changes, it will make it harder for you to plan the scope of your testing. If you do not know when, it is difficult to trust the result of your tests.

One way of identifying your system is to first identify what the system consists of. Considering the boundary of the system and what is included. Should you include configuration of the environment as part of the system? I would. Still, there are no perfect oracles. You will only be able to define the system to certain extent.

As you define parts or components of the system, you can also determine when each are changed. The sum of those components are the system and its version. I am sure there are many ways to do this. Whatever method you choose, you need to be able to refer to what it is.

I think it is extremely important that you do anything you can to explore what the system is and what possible boundaries it could have. You need many and different views of the system, creating many models and abstractions. In the book “Explore IT!”, Elizabeth Hendrickson writes about identifying the eco system and performing recon missions to charter the terrain, which is an excellent way of describing it. When talking about test coverage you need to be able to connect that to a model or a map of the system. By doing that you also show what you know are coverable areas. Another way of finding out what the system is using the heuristic strategy model, by James Bach, and specifically exploring Product Elements. Something that I have experienced is that when you post and visualize the models of the system for everyone to see, you will immediately start to gain feedback about them from your co-workers. Very often, there are parts missing or dependencies not shown.

If one of your missions as a tester is to inform stakeholders to make sound decisions, then consider if you know enough of the system to be able to recommend a release to customer or not. Consider what you are referring to when you talk about test coverage and if your view of the system is enough.

References

1. Explore It! by Elisabeth Hendrickson – https://pragprog.com/book/ehxta/explore-it

2. Heuristic Test Strategy Model by James Bach – http://www.satisfice.com/tools/htsm.pdf

3. The Oracle Problem – http://kaner.com/?p=190

4. A Taxonomy for Test Oracles by Douglas Hoffman – http://www.softwarequalitymethods.com/Papers/OracleTax.pdf