Subject: Re: Development Shop FSB
From: Tom Lord <>
Date: Tue, 12 Mar 2002 23:06:49 -0800 (PST)


   From: Ian Lance Taylor <>

   Tom Lord <> writes:

   > It's impossible for an outsider to accurately measure such things, but
   > I'm skeptical that RH's contributions to the public corpus of open
   > source software corresponds to a decent percentage of a $2,520,0000
   > expenditure.

   Not even a decent percentage?  Counting their contributions to the
   kernel, gcc, gdb, binutils, postgres, ecos?  I don't see it that way.
   If that's a quarterly expenditure, we're only talking about 60 people
   or so.  

There's a typo.  I put the commas in the wrong place.  The annual
figure for "Cost of Services" is $25,200,000 and the report says of
this number "Cost of services revenue is primarily comprised of
salaries and related costs".  That annual figure is obtained by
multiplying the Sep-Nov 2001 number times four, ignoring the fact that
it declined that quarter.

I think the typo probably effected your computation.  Assuming that
you read the number as a a factor of 10 smaller and figured it was a
quarterly number, applying your heuristic, it's 150 people.
Personally, I'd guess a number somewhere in between 60 and 150,
probably closer to 100, but then I tend to think salaries should be
higher than they typically are.

One of the ways in which it's "impossible for an outsider to
accurately measure" is that it isn't at all clear that all RH-labeled
contributions to the projects you mention are either (a) attributed to
that cost in the report or (b) attributed to any RH cost (as opposed
to including spare time volunteered by employees).

Even if all those contributions do come from just that cost, it's also
worth considering a distinction between "lines of code that wind up in
a public project" vs. "contributions to the success of the public
project".  For example, if you add lines of code to program FOOBAR and
those lines of code benefit hardly anyone who isn't a RH customer but
do add to the shared maintenance burden of FOOBAR, is that really a
contribution?  On the other hand, if you, say, improve the internals
documentation of the 10 trickiest files of FOOBAR, pick off 100 bugs
that nobody is paying (specifically) to have fixed, and make
systematic accessibility improvements to FOOBAR -- those are more
clearly contributions.  I don't see great quantities of the latter
kinds of contribution coming from Red Hat (some, sure, but not a
breathtaking amount) -- have I missed something?

While I don't wish to defame the contributions RH makes, my point was
simply that they don't seem to have O(100) people paid for out of the
services budget focussed squarely on the problem of improving the
public projects, even though such work would create value for
commercial users.  If your goal is "to do what I enjoy doing for a
living, develop Free Software" and what you mean by that is
"contribute to the shared source base in the areas where my skills and
interests can do the most good, working freely with whoever else is
working on the same projects", the RH Services business model doesn't
seem to create the kind of job you're looking for (with perhaps, a
small number of "super-star" exceptions).  I think a lot of useful,
fun, in some cases long overdue work doesn't get done because "there's
no contract for that" -- and that's a business model problem.

   Many of them are not genius programmers, of course; they're
   just journeymen.  

I wouldn't say "just" and I wouldn't overlook the genius of journeymen
programmers.  When I use that phrase, I simply mean programmers who
bring a particular specialized and repeatable skill to projects and
who generally are not the one's responsible for overall planning.  The
repeatability and accuracy of what they do, if they do it well, is

   Remember also that Red Hat does a respectable amount
   of testing work (at least, respectable compared to the amount of
   testing which anybody else does), which mostly shows up as minor bug
   fixes.  Considering the contributions I see publicly, I think there
   is a decent percentage coming out.

Reasonable people can disagree on this point.  It's interesting,
though, that you say RH does a respectable amount of testing -- but
that nobody else does.  Is it reasonable to conclude from that that
RH has testing software which they don't release?

If so much testing is necessary, and it is so expensive that nobody
else can afford to do it, then wouldn't we expect the economically
efficient solution to be the one that eliminates the need for such
differentiated in-house testing at its source: in the public projects?

Sure, a stream of bug-fixes is nice, but the cost to others of
incorporating them into the public projects isn't 0, and the cost to
others of not being able to prevent them in the first place (and
having to wait for RH to find them and send bug reports) is also not
0.  The processes RH uses to find these bugs could be externalized and
that would raise the level of the playing field.  In the mean time, it
seems to me to be pretty distorted that one company winds up being the
"magic testing cauldron".

   > It's worth the most when you instantly commoditize it by integrating
   > it into the public corpus.  RH brags a lot about the amount of
   > development they do but I don't see them even trying to integrate that
   > work into the public sphere except in cases where they can take more
   > from others than they give back.  Thus, they play in the "classpath"
   > and "gcc" projects, but hold back in the installation program and
   > upgrade-over-the-net spaces.

   As far as I know, Red Hat's update-over-the-net program is called
   up2date (I've never used it myself).  The sources are on the Red Hat
   web site and are included on the Red Hat CD.  I don't know about the
   installation program--I'm not sure what it's name is.

I wasn't talking about up2date, I was talking about the software
infrastructure behind this: "".