Subject: Re: who's running your business?
Date: Wed, 31 May 2000 17:24:54 -0400

Kragen Sitaker writes:
> I'm reluctant to publicly disagree with luminaries such as Ian, because
> I'm usually wrong when I do, but I'll do it anyway.

Not in this case apparently. :-)

> There are some kinds of bugs testing is very poor at finding.  Even
> exceptionally good testing --- meaning full code-coverage testing, plus
> spec-coverage testing (I'm not sure of the term for this) where you
> test each variable having each of the meaningfully-different sets of
> values it can have --- e.g. for division, you want to test cases where
> each of the numerator and denominator is positive, negative, and zero,
> plus stochastic use-case-based testing --- will be unlikely to find
> certain kinds of tests.

Absolutely.  Bugs are most likely to show up at boundary cases, and
testing after the fact is very unlikely to spot such boundary cases.

> Furthermore, such extreme testing is very labor-intensive.

You said it.  The problem scales exponentially with the amount of code
that you have.

> Reading code critically is likely to expose different sorts of bugs
> than testing it.  Doing both together is likely to expose more bugs
> than either alone.

What I have read is that doing simple code walkthroughs is more
efficient at catching bugs than testing is, and can catch them at a
point where it is far cheaper to fix.  Not to mention the benefits
in terms of knowledge transfer...

> All of this is well documented in that famous journal, _Somewhere In
> The Literature_.  :)

Make that _Code Complete_ by Steve McConnell.  If you have not done
so already, buy it by yesterday, read it by the day before that, and
then buy copies for your employees.  I am unsure whether I am kidding
about that timeline...

Not only does it back up the above statistic (sorry, my copy is at
home or I would give you chapter and verse) but it gives you
references to the original research papers as well.