Subject: Re: who's running your business?
From: Ian Lance Taylor <ian@airs.com>
Date: 31 May 2000 13:56:31 -0700

   Date: Wed, 31 May 2000 16:15:45 -0400 (EDT)
   From: kragen@pobox.com (Kragen Sitaker)

   I'm reluctant to publicly disagree with luminaries such as Ian, because
   I'm usually wrong when I do, but I'll do it anyway.

Yikes, don't let my confident tone mislead you.  I'm frequently
mistaken.

   There are some kinds of bugs testing is very poor at finding.  Even
   exceptionally good testing --- meaning full code-coverage testing, plus
   spec-coverage testing (I'm not sure of the term for this) where you
   test each variable having each of the meaningfully-different sets of
   values it can have --- e.g. for division, you want to test cases where
   each of the numerator and denominator is positive, negative, and zero,
   plus stochastic use-case-based testing --- will be unlikely to find
   certain kinds of tests.

   Furthermore, such extreme testing is very labor-intensive.

   Reading code critically is likely to expose different sorts of bugs
   than testing it.  Doing both together is likely to expose more bugs
   than either alone.

I think I must have said something I don't really believe, but I
didn't bother to save the mail so I'm not exactly sure what it was.

I don't think testing is the one answer to finding bugs.  In fact, I
agree with you that reading the code carefully is better than testing,
and that both testing and reading is better still.

I was trying to argue that proving code correct is not a silver bullet
either, and one of the problems with most proof approaches is that a
correct proof does not necessarily imply correct code, because the
proof is not the same as the code.  Understanding that the proof
correctly describes the code is not itself a trivial problem--it's not
all that much easier than actual programming, in fact.

As it happens, I believe that the most effective way to find bugs is
code reviews by other programmers.

My personal preference for testing is regression tests for ongoing
maintenance, rather than correctness tests.  My favorite tests (when
applicable) are things like the TeX torture test, which pump in a
whole bunch of input and generate a lot of output.  If something
changes in the output, then you may have found a bug in a recent
change.  Or you may have fixed a bug.  Either way, you know that you
have to investigate and understand the change.

   All of this is well documented in that famous journal, _Somewhere In
   The Literature_.  :)

I like ``The Psychology of Computer Programming,'' by Gerald Weinberg.

Ian