Subject: Re: improving project maintainership
From: "Forrest J. Cavalier III" <>
Date: Mon, 11 Feb 2002 13:05:18 -0500 (EST)

> There are two views on what business the big FSBs and semi-FSBs are in
> and really I think the truth is that they are in transition from A to
> B:

This business B sounds like a nice company to work for.  But your
analysis dismisses most of the barriers to competition that Business A
uses to maintain margin, and did not replace them.  How does business
B get revenue, and how does it maintain margin?

Assumably the only reason they are in transition is due to
market pressures.  Can you explain those pressures in
each point?

>         Business A:
>         * Collect a bunch of publicly licensed pieces that are hard
>           for your customers to assemble.  You may be shooting
>           yourself in the foot because if all you ever do is collect a
>           fixed number of pieces, eventually it will become easy for
>           your customers to assemble those same pieces themselves.

Even if/when assembly becomes easy for the customer, (presumably 
copying is already easy) there are still problems of QA and 
selection.  You say this exactly in your Business B part.  "RedHat"
means something to consumers in terms of QA and selection.  They
are paying for the "appropriate" collection, not the bits.

>           Still, in the short term, you are definitely adding value.
>           You could look for ways to direct a lot of your revenue
>           towards expanding the number of pieces -- but right now your
>           money is tied up in the other aspects of "Business A".

"tied up" how?  Hopefully everything in a business is paying its
own freight, so to speak.  If you mean that selling collections is
typically low margin for FSB, I agree.  But if the margins/market remain
adequate to support a business, there is no market pressure which will
make all such businesses dissolve.  So what is pushing Business A
to be Business B?

>         * Do some testing to decide which pieces to upgrade to and
>           which not.  You probably can't spend enough to do a really
>           thorough job at testing but you can somewhat make up for
>           that in a non-collaborative way by hiring some of the core
>           developers and aligning their priorities.  Those core
>           developers can focus and prune your testing efforts. This
>           may be shooting yourself in the foot, since those core
>           developers won't have as much time to enable collaboration
>           as well as they could have, but so long as you don't force
>           the issue, you can count on those developers to strike the
>           very best balance they can under the circumstances (until
>           they burn out or get rich and bored or become entrenched and
>           evil).

That seemed to ramble.  Can you be more concise?  In my observation
developers do a pretty good job at creating solutions, but it is rare
to have a good developer also good at testing and documentation.  
Do you mean to include market tests in order to decide future feature 
sets? If you mean "develop" where you used "test", then I can see this point.

But for reasons I tried to get at in a different message,
this is going to be part of business B because testing is
hard, and you have finite resources.

>                 You could help eliminate much of the testing expense
>           by helping to make it more natural for the public projects
>           to maintain the testing infrastructure themselves -- but
>           right now your money is tied up.
>         * Do a decent amount of release engineering.  If your
>           distribution is popular, you can even get volunteers to do
>           release engineering for you. if you (and your customers) trust the volunteers, or have
some way of  cross-checking everything important they have done.
(Sometimes this is easy: "Does it compile."  Sometimes it is very hard
"Does this reliably work with video card X, and printer Y.")

>   Again, you may be shooting
>           yourself in the foot because, together with your
>           competitors, you're just wasting in-house and volunteer
>           effort as you partition employees and volunteers into those
>           doing release engineering for your distribution vs. those
>           doing release engineering for other distributions.
>           You could eliminate most of the expenses of release
>           engineering by investing in tools that make it easier for
>           the public projects to do release engineering themselves --
>           but right now your money is tied up.

To engineers, avoidable waste is like a plague: eliminate
it first.  But does waste in the system as a whole create sufficient
market pressures?  Producers have pressure to eliminate waste
because it helps margin.  Consumers sometimes buy products and services
to eliminate waste.  (There are other reasons they buy.) 

But what real market pressure is there to improve efficiencies (aka
prevent waste) of third parties?  

Maybe this is why you were thinking in terms of taxes, and
really did mean to use a legislated burden to pay for improvements
presumably benefiting everyone.

I'm not a strict libertarian, so I don't object to that on
principle, but I don't think such a tax will ever be enacted.

>         * Build an installer and an upgrade-over-the-net tool.  As
>           long as you're fighting with competitors over release
>           engineering technology, you'll certainly want to have unique
>           solutions in this area -- especially solutions that can
>           subsume your competitors work.  Installers seem to be the
>           dominant topic addressed by reviewers and, presumably, by
>           equally shallow-looking purchasers.

"Shallow"?  Consumers buy based on what is important and impacts
them, for good or bad.  I'd say install/upgrade is pretty
important, not shallow.

>                                      Upgrade-over-the-net
>           tools satisfy an IT dept's quest for lower cost of ownership
>           and your quest for "ties". 

Again, this focuses on the producer side of this equation.
The consumer side is what matters.  Upgrades generally have low
transaction costs.  Switching suppliers is expensive.

                                  Sure, a public project can come
>           along, produce standard solutions in these areas and blow
>           your effort out of the water, but until they do, isn't it
>           rational to keep your solution as uniquely yours as
>           possible, even if that means forsaking the benefits of
>           collaboration on its development and extension?  

See above. The consumer side is what matters.

>           none of the big hw vendors would have incentive to drive a
>           standard solution in these areas: one can hardly imagine HP,
>           IBM, and Sun finding a way to collaborate on an issue like
>           this.

Any one of those companies could set a few people on it and their
bottom line wouldn't be any worse off.  I can easily imagine it,
simply because it is so insignificant to them.  They do sillier things 
without identifiable reason.

>                  Besides, isn't your branded solution in these areas a
>           way to leverage your specific release engineering
>           technology?  And anyway, just now all your money is tied up.

You keep "tieing up money" in business A, but don't do that for 
business B.  Looking longingly at huge cash flow as "tied up"
money is not useful, because you can't just decide to "free"
any of it for other purposes.  Cash flow results from your
business processes, it isn't something you can re-allocate.  

>         * Develop non-free documentation.  Look, its a revenue stream
>           right?  And its necessary given your (effectively)
>           proprietary install/upgrade technologies.  Sure, you could
>           hire or free-lance writers to produce public documentation,
>           taking care to develop standards for structure, formatting, and
>           installation.  Doing that would allow more people to help
>           develop the software.  But for now it gives you a
>           competitive advantage.

The typical end-users don't care about documentation.  Used to be 
that if you paid > $100 for mass-produced software, you were
outraged if you didn't receive many pounds of dead trees inside.

Today most spend hundreds of dollars and get some $0.50 CD's and
are happy, never even using online help, much less printed docs.

There are sub-markets which are different.  Programmers still
make very extensive use of documentation, for example.  And
the quality of the documentation matters a lot to them.  It
can save or create hours or days of work.

Corporate Help desks, trainers, and the like need good docs, too.

>         * Develop new markets.  Ok, you have very few (but not 0)
>           opportunities to develop new components.  Coming up with and
>           supporting a new configuration of existing components is
>           hella expensive.  So, you've got to be shrewd and lucky to
>           identify just those markets that are close enough to reach
>           but large enough to be worth reaching.  

You admit that developing new markets is in Business B too.

>         Business B:
>         * It's easy to put together the publicly available pieces.
>           What's hard is picking the pieces to put together to 
>           achieve particular goals.  It's hard because there are many
>           solutions and evaluating them is a complex task.  That's
>           part of your job.

Business A, too.  What contrast are you making?

>         * A lot of testing happens "for free".  Every significant
>           package has an extensive test suite.  The GNU distros or the
>           public maintainer distros (and candidate distros) are farmed
>           out over a P2P network that automatically does a lot of
>           testing on many different platforms.  For weird
>           configurations that only your customers want, you run those
>           same testing tools in house, very inexpensively.  Your
>           larger customers run those same test in-house for confidence
>           building and risk assessment.  You have a little server farm
>           that contributes to the P2P testing network.

I think that is far removed from present reality and tools.  I don't
disagree that there is a lot of testing for free.  Posting volume
to bugtraq@securityfocus seems to indicate we have a long way to go.

Who gets the egg on their face if all that free testing missed the
defects important to your customers?

Testing will be essential and non-trivial until system specs, 
interfaces, and software are written defect free.  That's a
LONG way off.

>         * Release engineering happens "for free".  Every significant
>           package conforms to the GNU coding standards version 121.
>           Those standards have enough features that the path from
>           maintainer FTP sites to your CD-burner or OEMs is automated.

You mean the October 2001 GNU coding standards?  They don't have
anything close to what most would consider adequate quality
assurances.  (But I think this point is essentially close to
achievable reality, IF you get all the maintainers to comply with
adequate standards.)


>           Some of those same employees
>           participate in customer working groups to develop the goals
>           and communicate where the architecture is heading.  With
>           some indirection the same pattern occurs with customers who
>           are individual consumers buying toys, communication tools,
>           and home management tools.  Your customers pay you to 
>           participate in those working groups.

Customers pay you to solve their problems.  They aren't going
to object when you decide that participating in a working group 
makes sense, but they aren't going to sponsor you directly.  
Once the working group has produced a standard, the playing
field quickly levels, bringing all non-participating competitors
up to the same level.