Subject: Re: is there a statistician in the house? (long)
From: "Stephen J. Turnbull" <stephen@xemacs.org>
Date: Fri, 11 Mar 2005 13:46:55 +0900

>>>>> "Seth" == Seth Gordon <sethg@ropine.com> writes:

    Seth> (1') Another way of stating this is that there is no global
    Seth> ordering for people's reputations.  If I say that A is a
    Seth> more praiseworthy hacker than B, and you say the opposite,
    Seth> there is no yardstick for judging between us.  Of course, if
    Seth> we both are trying to hire a programmer and we start putting
    Seth> out competitive bids, we will discover whether or not A's
    Seth> labor has a higher *market* value than B's, but then we are
    Seth> no longer in the realm of a reputation-based economy.

    Seth> (2) Reputation is ordinal, not parametric.  That is, I might
    Seth> say that A is a more praiseworthy hacker than B, but I
    Seth> cannot quantify *how much better* A is.

And as you point out, the market has solutions to both problems.  That
is an existence proof.  So shouldn't we be looking for a global
cardinal measure of value, rather than trying to aggregate local
ordinal measures?  (Note that this is not advocating the market, this
is advocating a search for a solution which has good properties which
are known to be feasible.  "Reputation" as you describe it doesn't
have them, though.)

    Seth> (5) When you are deciding who to favor with gifts in the
    Seth> current turn, you let yourself be guided by the rankings in
    Seth> both your primary and secondary rank lists.

    Seth> Comments?

First, I think that (except for the use of Perl modules), this is an
accurate representation of how gift economies already work, and
specifically the open source community.  I know that I put a lot of
weight on the opinions of people I respect and especially those I know
personally when evaluating reputations of other hackers, both famous
ones and not-so-famous ones.  I explicitly solicit them in many cases.

But I doubt I would get better results by doing the formal statistical
tests, especially since I can and do incorporate cardinal information
in my informal cogitations.  Cf.
http://turnbull.sk.tsukuba.ac.jp/Teach/IntroSES/ohp54.html and
following couple of slides.

Ie, A is marginally better than B who is twice as good as C means that
C is out of the running no matter what my secondary ranking looks
like, while I would surely award the gift to B if somebody I respect
said "IMHO, B is _significantly_ better than A."  I observe others
doing the same, yet it would be hard to formalize that procedure.

Second, your mechanism fails to address the two fundamental problems
that gift economies generate vis a vis open source software:

    1.  The resources devoted to gifts to open source developers are
        too small compared to the benefits received from them.  (In
        the sense of society-wide economic efficiency.)

    2.  The aggregation scheme gives no systematic weight to small
        contributions to 100% of the population vs. slightly larger
        contributions to tiny minorities.

#1's meaning is obvious.

Another way to phrase #2 is "fragmentation of the gift economy into
small reciprocal cliques".  Statistically, the problem is that the
endorsee relation is likely to be symmetric, and evaluations within
cliques homogeneous.  Cross-clique endorsements will be less common,
and likely to be statistical outliers.  (I suspect the Kruskal-Wallace
test will tell you so, too.)

Third, I rather suspect that your procedure would turn out to be easy
to "game".  To give a clear example, I like A and you like B, our
actual rankings are A, C, D, E, B and B, E, D, C, A, and we agree to
post rankings of A, B, (C, D, E) and B, A, (C, D, E) respectively.
While I don't expect that many FLOSS people would actually game the
mechanism that way, I think there would be strong incentive to
"campaign" for your favorite developers if the mechanism were
formalized in this way.  Currently "campaigning" won't work in the
absence of real contribution widely known, as the information that
campaigning was happening would be incorporated into one's informal
evaluations.  But if the aggregation were to be formalized as you
proposed, knowledge that the rankings may have been influenced by
active campaigns for A and B at the expense of C, D, and E would not
be part of the decision process.

-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.