Is your testing saturated? Rikard Edgren
There are many names for software testing strategies/activities/ approaches/processes; they can be risk-based, coverage-focused, exploratory, requirements-based, Super-TPI, TMM 5 et.al.
The names generally come from how the testing is performed or initiated, so I thought we should look at it from another angle, from the end of testing, from the results that we might know about a year later.
I like the content of Good Enough Testing, but the words are easily misunderstood; we don’t have definitions for Crappy or Decent Testing, and since that often isn’t enough it might be useful with Saturated Testing. (This is elaborated elsewhere with a different name??)
The term is borrowed from Grounded Theory (social science qualitative analysis), and means that further research in an area doesn’t give any important, new information, and therefore isn’t worth the effort.
The same can happen in ambitious software testing (if the test time isn’t radically shortened…): when important bugs no longer are found, and all important test areas have been covered, and no code changes are made, there is no need to continue testing.
I see three hypothetical levels of Testing Saturation:
1) more testing will not find information that would change the release decision
2) more testing will not find issues that later would be patched
3) more testing will not find bugs that we would like to have fixed
If I you use a mix of these as your goal, I have three warnings:
* Have you used all relevant information?
* Have you tried all relevant test approaches?
* Have enough people with different views been involved in (at least) generating test ideas?
To get a result you are satisfied with, you need to think about things like this early, you need a lot of skill and hard work, and a fair share of luck.
“Saturation” is a nice striking word.
The first thing I think about is the lumpiness of the coverage. You may feel that your product is saturated with testing, while in fact part of it is soaked, while other parts are parched.
Thanks, those are very nice additions to the vocabulary.
That’s the good thing with more words for the things we do, one starts to think in new directions, and sometimes you understand things a lot better, and/or can explain them better.
Lumpiness is something you’d want to avoid.
Strauss/Corbin: Basics of Qualitative Research page 212:
“The general rule when building theory is to gather data until each category is saturated”
When transferring this to software testing, we could imagine a matrix of features/use cases vs. quality characteristics. Each point that is deemed important should be appropriately “soaked”.
With the “right” granularity, this might be feasible, and could be a powerful item in a quality status report, especially if visualized in a Heat Map or a Treemap where size is set by importance.
Interesting idea.
I think this is important for many aspects of testing. When doing test planning where you dig into an area to identify test ideas you will know saturation level if you are collaborating with others and get feedback often on your work. In testing you can use the concept of test sessions as well the time limit on focus/defocus with a debriefing. This will let you, by collaboration, know when you get saturated.
The key for good testing and test planning is collaboration in the team to identify when you start to get saturated.
I LIKE this terminology. I got into a dispute a couple of weeks ago around that had a similar feel. A co-worker was unhappy that certain information wasn’t being considered. My position was basically that he was being too granular– considering the information (that a test which was failing on some platforms was passing on other platforms) wasn’t relevant to the particular decision we were trying to make. We already had enough information (that the test was failing on at least one platform) to decide what our next action was.
Are the situations in which a particular test suite/tool/process could be saturated?
We have some data-driven automated tests that take several days to run through the entire data suite. The time between the builds being published and the test results becoming available was devaluing the results of the tests. We broke it up into multiple suites, including a build validation suite that finishes in about 30 minutes.
Not sure if these ideas match up or not.
I tend to use the same thinking as you did Pete; there is no need to spend time trying to break something that is already broken (although sometimes you need to do just that in order to find out more about the impact, and about underlying/side-bying issues.)
Regarding saturation for your test tool, the match doesn’t seem perfect to me.
You rather did a good split-up to add value.
If you noticed that adding data to your tests didn’t provide any new information, also in the long-run, you could talk about saturation.
On the other hand, if the tets doesn’t cost maintenance, it’s good to run all of them, once in a while.