Subject: Re: a tool free software developers need
From: Ben_Tilly@trepp.com
Date: Fri, 15 Feb 2002 15:17:36 -0500


Tom Lord wrote:
> Here's a more concrete example of something to build as part of a
> transition to a business "B" world.
>
> Lately, I'm getting bug reports that are very platform and
> configuration specific.  These aren't necessarily obscure systems --
> more like "RedHat 7.2 with Bash 2.05a.0 and gawk 3.0.6" or "Solaris
> mumbledefoo on an Ultrasparc" -- but they are different from the
> systems I have and at this stage, I usually can't reproduce the bugs
> reported on the machines I have.
>
> Sometimes the bug reporter is willing to act as a (very low bandwidth,
> very high level) remote debugger; sometimes not.  Sometimes they have
> the skill for that; sometimes not.
>
> I could go a lot faster if I could get as accurate a description of
> the platform as possible, fire off an email to a friendly platform
> farm, and get back a reservation:

For a generally accurate description of the platform I would suggest
having them run:

  perl -V

The output of that is intended for use by Perl developers in sorting
out bug reports (it is, for instance, automatically included in the
output of perlbug).  This has been *extremely* valuable over Perl's
lifecycle.

As for auditing of code on many platforms, the Perl world has been
inspired by Mozilla's tinderbox and has taken steps like that for CPAN
with, I understand, some degree of success.

>    To: lord@regexps.com
>    From: friend-platform-farm@osdl.org
>
>    You have a 3 hour reservation on test235.odsl.org at 3am
>    on 2002-02-16.  Your ssh key is attached.
>
> Presumably "test235" would be freshly loaded with a
> closest-available-approximation of my platform spec when I log in, and
> would be re-loaded after I leave.

This doesn't exist, but several times now I have seen reports of vendors
making machines available for this kind of testing.

Within the Linux world I think that something like this could be done
with user-mode Linux.  In the Intel world with VMWare.  In general,
hrm, um, would be nice...

> The platform farm would also be handy if I could automate my end --
> simply have it pick release candidates from my repository and run a
> test suite, mailing me the results on a number of platforms.  There
> doesn't have to be just one platform farm: it might make good sense to
> build a P2P network so that trusted volunteers from all over can throw
> machines into the pool.

Check out the tinderbox effort from Mozilla.  To make this available
for untrusted code from random developers you would need some very
good sandboxes though.  Else people won't want to volunteer their
machines for would-be crackers.

> Implementing tools like that would also make it more likely that
> various projects would raise the priority of beefing up their test
> suites -- because there'd be a direct pay-off for doing so.  It would
> be an opportunity to widely deploy (de facto) standardized frameworks
> for building, testing, and installing packages.

I think that education would raise that priority.  The simple act of
encouraging the standard:

  perl Config.PL
  make
  make test
  make install

paradigm has done Perl wonders.  Even if people usually do it with:

  > perl -MCPAN -e shell
  install Foo::Bar

Perhaps make s/Even if/Particularly if/ since what is not done
automatically is not done reliably...

Cheers,
Ben