Subject: Re: Reuse - is it for real?
From: "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>
Date: Wed, 27 Oct 1999 17:29:07 +0900 (JST)

>>>>> "Crispin" == Crispin Cowan <crispin@cse.ogi.edu> writes:

    Crispin> Open source developers have a MUCH larger pool of
    Crispin> software from which to draw than closed source
    Crispin> developers.  Further more (and this is important) there
    Crispin> are many more open source developers with access to this
    Crispin> vast library than there are closed companies that possess
    Crispin> a vast library.  Finally, the number of times the wheel
    Crispin> gets re-invented is linear in the number of would-be
    Crispin> closed companies, while wheel re-invention is ~=O(1) in
    Crispin> the open community.

Look who's doing ivory-tower theorizing now!  The friction involved in
finding the right module _outside_ of your specialty is large (you
point out that specialists tend to cluster in open source as they do
in closed source vendors); that large pool exists but may not be
relevant, and probably is not used much.  In fact, you admit that
yourself later in response to a concrete example.  O(1) is hopelessly
optimistic.

Closed source vendors, on the other hand, have an option not available
to open source projects: buying modules from closed source vendors.
They also have access to the open source library.  They may not be
able to use it as conveniently as the open source projects can, but it
will be helpful in many cases.  O(n) is slanderously pessimistic.

I tend to think that the main differences between open source and
closed source models with respect to reuse are (1) cherry-picking
reuse (I need a Foo that does Bar <hack, hack, hack>), (2) enforced
modularization by bazaar development makes it cheaper to produce
reusable modules, and (3) the financial resources available to
monopolies _can_ be applied to investing in reuse.

Also, w.r.t. open source, I think the facts motivate against you.  A
few examples:

  o Most Perl modules have parallel Python modules, and often are also 
    reimplemented in C, ruby, slang, pike, tcl, whatever.  In general, 
    people prefer to use their favorite language, and if it doesn't
    have the feature they want, they reimplement it.  (Once we have
    lots of compile-to-java scripting languages, we may be able to mix 
    and match, but I won't bet on that.)
  o Most device drivers for Linux are maintained independently of
    those in BSD AFAIK, despite being written in C.
  o Although Emacs does not share XEmacs code for reasons of legal
    paranoia (large portions of XEmacs code are copyright Sun, and Sun 
    consistently refuses to respond to requests to assign to the FSF), 
    microemacs et al do not share code with the GNU Emacs, and John E
    Davis went so far as to invent a new extension language, with the
    effect of preventing extension-sharing.
  o Most DnD code has to be written to deal with multiple protocols
    (although maybe Linux just has one? I think not).
  o EsounD vs. rplay, Xaudio, NAS.  (I'm guessing, I don't know
    EsounD's pedigree but the characteristics make it likely.)
  o AFAIK, ditto the various awks, ditto the various makes, ditto....

It's hard to say, of course, since the kind of reuse you rely on is
indistinguishable from bazaar development.

    >> o Random hacking is not likely to be directed toward user
    >> needs, but rather toward personal needs (nothing wrong with
    >> that, but it means that external benefits to Open Foo may be
    >> only marginally higher due to this effect).

    Crispin> Disproving this assumption is part of the grand
    Crispin> experiment.

It's not an assumption, it's a theorem, derived from two assumptions.
(a) Hackers have personal needs which they act to satisfy.  (b)
The distribution of needs among hackers is not identical to the
distribution of needs among the general user population.

Whether it really matters in practice is a matter for experience.

I believe that hackers are different.  For empirical evidence, I put
it to you that Emacs is not Word.  Thank Heaven!

    Crispin> * Open source approach: I really like the Foo
    Crispin> application, but I wish it had a Bar feature. <hack hack
    Crispin> hack> There, now it has a bar feature.

    >> The standard objection to this in the management literature is
    >> that it doesn't scale.

    Crispin> The standard management literature seems to be getting
    Crispin> trampled by the onslought of open source development.
    Crispin> The rules have changed.

Oh, brother, do _you_ have big surprises waiting for you.  IMHO.  I
Hope Your Mileage Varies.  :-)

    >> I can say, personally, having a lot of experience with a
    >> particular Bar feature (Japanese input/output/massaging
    >> capability), that some Bar features added by this approach

    Crispin> The problem of integrating Asian character sets into open
    Crispin> source products is IMHO a worst-case scenario.  A vast
    Crispin> majority of open source developers are barely aware of
    Crispin> the Asian character set problem, so it gets very little
    Crispin> attention.  So little that products are often designed
    Crispin> without any allowance for Asian characters, making life
    Crispin> even tougher on the few who try to address the problem.
    Crispin> This is an instance of a *lack* of common interest across
    Crispin> different segments of the community.

Thank you for explaining the mechanism so well.  It may be worst-case,
but it's not so much different from many common issues.  That's why
there are "segments" of the community.  Coordinating those segments is
costly.

    >> o waste effort because the Bar programmer doesn't care about
    >> Foo's design or coding standards, resulting in a new patch with
    >> every re-release of Foo

    Crispin> And people are aware of this problem, and often work to
    Crispin> avoid it by coding their enhancements with the intent of
    Crispin> having them integrated back into the original product.

Maybe.  This one has to be proved by experience.  The Japanese
experience, although exceptional, is quite the opposite.  The only
major _enhancement_ produced for Japanese integrated into the
mainstream I know of is Mule for Emacs, and that took _ten years_, and
a full redesign to radically different specs.  The TT X servers may be
another counterexample, but I know there are a couple of versions out
there and I don't know that the Japanese version is the winner.  The
Japanese-built VFlib servers certainly have not made it into the
mainstream.  I know of dozens of patches that never made it into the
mainline and die.  Occasionally one is revived and limps along for a
while.

The Japanese experience is important because it does show that if the
core developers don't care about the issue, free software is quite
capable of failing to address great inefficiencies.  I don't know that
proprietary software would do a better job (although everything
Microsoft sells has been ported to Japanese AFAIK).

    >> Well, anybody who is thinking about extending XEmacs, my advice
    >> is "think twice".  _None_ of the major low-level components
    >> (redisplay, Lisp interpreter, Mule) is close to
    >> state-of-the-art.

    Crispin> Programs do age and die, at which point they need to be
    Crispin> thrown out and re-done.  Emacs is near 20 years old.
    Crispin> During that time period, gcc has been re-written from
    Crispin> scratch twice.

The exception that proves the rule.  When Linux and gcc stop getting
rewritten from scratch every five years, we can consider the movement
dead.  Every hacker has ambitions to contribute to those projects,
even though under today's conditions few are competent to contribute
substantial blocks of code.  I would be surprised that if there wasn't
that much interest.  In fact, I'm surprised that GNU Emacs has only
been rewritten (partly) from scratch once in that period, despite the
agony of the fork.  I think that's clear evidence of a shortage of
hacker time.  :-)


-- 
University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
__________________________________________________________________________
__________________________________________________________________________
What are those two straight lines for?  "Free software rules."