Subject: *precisely* a commons (with tragedy)
From: Tom Lord <lord@regexps.com>
Date: Fri, 4 Jan 2002 03:44:56 -0800 (PST)



       Karsten opines:

       Tom:
       > The bits in a distribution are not rivalrous.  The "commons" is the
       > engineering infrastructure which produces, maintains, and deploys the
       > software, not the software itself. =20
       
       You're not being clear.  This engineering infrastructure isn't a
       commons.  An arbitrary user can't lay claim to my own resources.  An
       employer or client can hire my time.  I can volunteer it.  

Your comments reflect a deep misunderstanding.

The time or labor power of engineers is not the engineering
infrastructure.

A software engineering infrastructure is a mixture of processes,
information repositories, and information flows.[1]  It requires an
investment of engineer-hours to sustain or improve an engineering
infrastructure, but those engineer-hours are not the infrastructure
itself.

A software engineering infrastructure is a consumable resource.  The
ways in which it is consumed are numerous and subtle, but to choose
just one easy-to-understand example: the bandwidth for patches
(contributed proposed changes) to a particular project is finite;
scheduling the allocation of that bandwidth has to look beyond simple
fairness-of-access or merit-of-proposed-changes to consider factors
such as avoiding forking a project.  Therefore, a busy contributor
with the resources to fork a project can consume an unreasonable
amount of that bandwidth, excluding or slowing down the processing of
contributions which may very well be important to the long term health
of the project.

A software engineering infrastructure is a renewable resource.  It can
be renewed by feeding it engineer-hours.  It can be renewed by
implementing process improvements and by building and deploying new
software tools.

For public software, developed by open source processes, the
engineering infrastructure transcends any particular corporation or
engineer.  It is distributed.  Components can migrate.  Roles for
participants are typically fillable by many people.  The engineering
infrastructure is a "virtual entity" which we hold in common.

Because it is a consumable, renewable resource which we hold in
common, the public software engineering infrastructure is *precisely*
a commons.

Proprietary software companies have a walled-off commons.  Their
engineering infrastructures *also* transcend corporate boundaries --
but their infrastructures are sealed-off from the general public by
trade secrets, licensing terms, etc.

The long-standing claim made by FSBs is that the mechanisms which
wall-off the proprietary infrastructures create both significant
inefficiencies and significant risks to customers.  In support of
"inefficiencies", they point out that with a public infrastructure,
there are more chances to spot and act upon opportunities to improve
programs.  In support of "risks", they point out that customers of the
public infrastructure can inspect their goods and avoid lock-in.

Unfortunately, the predominant FSB practice has been to exploit the
public infrastructure without making a correspondingly large
investment in its renewal.  They make some investment, but not enough.

What would we predict that an over-grazed public software engineering
infrastructure would look like?  We'd look for promising projects that
stall or even die.  We'd look for patches of still-healthy landscape
that are over-crowded with consumers (e.g. lots of FSBs trying to sell
essentially the same things, rather than spreading out over the
landscape).  We'd look for lots of corner-cutting in the popular
engineering practices (e.g. shoddy documentation, weak testing
practices, weak tools, absense of external review).  We'd look for
post-apocalypse-style gang warfare and turf fighting.  We'd look for
entire, large sectors of the software economy in which FSBs are
willfully inactive because they recognize the resources aren't there
to compete.

And that's pretty much what we've got.  (There's a slightly more
optimistic assesment at the end of the message, though.)

I wrote:

	The problem seems to be that FSBs are selling what they know
	how to sell, and what their customers know how to buy and
	receive, rather than what they ought to be transacting over.
	For what they *are* selling, prices are naturally low, so the
	engineering infrastructure becomes "over-grazed".

I stand by that.

Proprietary companies charge much higher prices for more or less the
same kinds of products, obtaining the higher prices via a combination
of licensing and software license management.  They turn around and
invest more in their walled-off engineering infrastructure.  The
engineering infrastructure, with its feedback mechanisms between
engineers and customers, becomes the real product.  The legal
arrangements and license managers just a formal dance the participants
go through to facilitate the flow of money.

There is a natural competition between each family of proprietary
companies and the collective of FSBs.  From a shallow perspective,
each of these groups offers their prospective customers a set of
programs -- and we can just compare programs to see whose is better.
But from a deeper perspective, taking into account that software
evolves and that separately sold components synergize, it isn't sets
of programs that are competing: it's entire software ecologies.

FSBs have started small and when they've grown, its been by finding
and filling niches somewhat nimbly.  Because they're small, they seek
out niches for which the public engineering infrastructure is a
significant help: they graze heavily.  Confronted with skepticism
about the public software, they won customers with very low prices.
Consequently, their margins are now low, so dipping into the bank to
start renewing the commons is a hard decision to make.

The result is a chicken-and-egg problem.  Given a shaky looking public
software engineering infrastructure, and relatively small FSBs, its
pretty hard to make the case to customers that you now want to start
charging prices comparable to what a MS would get.  If we already had
a thriving, complex public infrastructure, making the case that a high
price is justified to represent a customer in that infrastructure
would make obvious sense.

I think the niche successes of FSBs justify very large injections of
cash now, both from investors and from customers conscious of their
long term needs.  FSBs have proven the efficiency of open source
processes, especially surrounding the kernels and web servers.  Those
parts of the software infrastructure are holding their own at the
moment and do receive good levels of renewal from FSB.  The primary
threat to those areas isn't that they themselves will collapse from
overgrazing, but that they can't survive in the long-term as an oasis
in an otherwise burned out landscape.  (Another threat is that the
turf battles over those tiny patches can be ugly and brutal and people
get badly hurt in them.)  Its a good time (well, quite a few years
overdue, actually) to start throwing cash at the problem of seeding
and tending the surrounding landscape.

-t


[1] Given more time, I might come up with a more refined definition,
    but that one is at least a first approximation.