Automated random or fuzzy testing by random input Martin Jansson
Random testing or fuzzy testing is nothing new, but for those of you who are new to it I just wanted to share a little tool I found. If you want to know a bit more about fuzzy testing go read at http://en.wikipedia.org/wiki/Fuzz_testing or whatever place you like to find quick info at.
Barton Miller has written a bit about this and made some binaries to use at:
ftp://ftp.cs.wisc.edu/paradyn/fuzz/
Read ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-nt.pdf for interesting MS Windows results.
The binary subjects an application to a stream of random input either by keyboard and mouse events or by using SendMessage and PostMessage. I used the binary on WinXP on UltraEdit and MS Word. I even tested the feature to simulate mouse click and mouse movement with a few interesting results.
What does this tell me? What would I expect from this? Is this a tool that I want to have in my tool box when testing software?
Using this tool will help you check robustness of an application. It will do automated random testing. If an application crashes often it might be that they have not properly handled Win32 messages. I will try this out in my toolbox. Even if you decide not to use it, read Bartons article.
Could you elaborate on your thoughts regarding:
“What does this tell me? What would I expect from this? Is this a tool that I want to have in my tool box when testing software?”
For me it is always a thrill to find errors with any cool method that triggers them. But the real coolness comes from those errors that are triggered with a cool method _and_ the errors are serious (or matters).
I tested fuzz.exe without doing too much investigation on what the result was. I noticed a few things, but it is always hard to be too sure what the actual result is from a test, especially automated.
I had one opion about how this worked before reading the article. Exploring fuzz.exe gave me some hints and some ideas that I got better context about after reading the article. For instance, the difference between PostMessage and SendMessage. There were more crashes using one of these methods. But most applications would crash if you tamper with them in an “illegal” way, was this what the tests were doing? I figure they were not after a while, but instead just using “legal” ways to tamper with the message handling.
Using this tool will cover just a few minor tests, still they are cheap and quick to perform. That is why I want to add them to the tool box.