Fuzz Testing

Development
1

I think in general we need to add some testing framework –
particularly to make sure we don’t re-introduce old bugs. I think we
also need ways to stress-test the core and each plugin. Marcus said he
had been looking at the QTest framework, so I’ll let him chime in here.

Today I tried out a Mac version of fuzz testing:
http://pages.cs.wisc.edu/~bart/fuzz/fuzz.html

My test script would launch Avogadro, give me ~2-3 minutes to set up
an initial state, and then start firing random keypress and mouse
events at the program. I set it up to send 25000 events with no delay,
and some events were sent simultaneously.

The results were instructive. On ~11 runs, I got about 8 crashes. One
run, the “quit” signal was sent. One run seemed to be OK. Two of the
crashes actually seem to be Qt-related (or at least Qt/Mac). One of
these was an assert and I didn’t get any crash info. So that leaves us
with 7 Avogadro crashes. (7/11 63% = bad.)

I’m attaching the Mac backtrace for each crash. The PrimitiveModel
definitely seems to need work. The IsoGen code also needs some work.

Cheers,
-Geoff

2

Hi,

very good idea with the testing;

by the way,

On Friday 20 June 2008 01:17:35 Geoffrey Hutchison wrote:

One of
these was an assert and I didn’t get any crash info.

then you can remedy that by running avogadro inside gdb, then you can get a
backtrace also after an assert has failed.

Cheers,

Benoit