Subject: Re: is there a statistician in the house? (long)
From: "Karsten M. Self" <kmself@ix.netcom.com>
Date: Thu, 10 Mar 2005 12:40:35 -0800
Thu, 10 Mar 2005 12:40:35 -0800
on Thu, Mar 10, 2005 at 12:04:03PM -0800, Ben Tilly (btilly@gmail.com) wrote:
> On Thu, 10 Mar 2005 11:56:43 -0800, Karsten M. Self
> <kmself@ix.netcom.com> wrote:
> > on Thu, Mar 10, 2005 at 02:25:20PM -0500, Seth Gordon (sethg@ropine.com) wrote:
> > 
> > > (1') Another way of stating this is that there is no global ordering for
> > > people's reputations.
> > 
> > Doesn't Google's PageRank refute this?
> 
> No.  Google's PageRank is a ranking of pages, not people, based on
> semantic information.  

Forest-trees impedence mismatch, Ben.

Google's pagerank uses inter-page references to draw inferences about a
page's overall relevance.  It's an interative and intensive calculation.
It  assumes  a relationship betweeen references and relevance.

A distributed trust system, say, PGP/GPG key signatures and trust
metrics, could be used as the basis of a similar system.  The nice thing
about Google is that it uses extant and behavioral information (people's
propensity to link pages) rather than requiring specific requested
inputs.  The result is far more data, though perhaps slightly weaker
correlation.  In balance, Google apparently thinks this is more
cost-effective than hiring a thousand monkeys to rate different pages.
With GPG, key signings are reasonably prevalent, particularly among
technical communties.  They're not a perfect predictor, however.

I'm not saying it's *practical* to do so.  Adoption of PGP/GPG is pretty
thin -- a friend reported a global keyspace relationship graph only
including ~250k keys.  This based on  public  keys (on keyservers)
 with  signatures -- so the overall adoption rate is likely higher by
some Finagle factor, let's assume between 2 and 10.

I'm also not saying the inference is valid.  Pagerank has its problems.
It's subject to gaming.  It's been tweaked significantly since
inception.  Etc.  But that's not relevant to the theoretical question:

    It *is* global and ordinal.



> People are not pages, nor is the semantic information embodied in
> human interactions readily captured or analyzed.

Yes, but that's a separate question.

Actually, it *is* regularly captured and analized (hint:  I can Point
you in the direction, but you have no Choice about your data being
present or accessed).  It's also pretty complete in some areas (credit
risk, say).  But not others.  And it's not  freely  globally accessible,
for better or worse.

Hint to all:  the problem of reputation assessment is  not  new.  It's
also not trivial.

 
> What web page, for instance, would you point at to figure out what my
> personal rank should be?

    http://www.experian.com/
    http://www.transunion.com/
    http://www.equifax.com/
    http://www.nsa.gov/

Not perfect.  But lots of measures, some of which would probably prove
statistically useful.


Answering another issue raised by Seth, markets  do  address this
question as well, though not on perfect information.  Market efficiency
is stymied by both thin data  and  thin trading, here.

This is also an issue somewhat addressed by the Jakob Nielsen approach.
Rather than taking complex measures of consituents of websites, fonts,
colors, et al, he uses  outcome measurement  --  time and success in
completing an assigned tast -- to measure site usability.  It's a
painfully simple approach.  It's also highly valuable.

Problems in reputation management systems generally run into the
following traps:

  - Defining what's being measured.  Reputation and/or competence in A
    is not transative to B.

  - Skill of the evaluator.  Rater A may have far less competence to
    judge than B.  How do you weight their inputs?  How do you assess
    their judgement?

  - Tuning to test.  Any quality assessment tool must of necessity
    measure a small number of quantifiable aspects.  Over time, testees
    tune themselves to maximize test results.

  - Corruption.  Attempting to influence either test-makers or
    evaluators.



I'm strongly inclined that anyone who wants to look at quality metrics
re-read  Zen and the Art of Motorcycle Maintenance .


Peace.

-- 
Karsten M. Self <kmself@ix.netcom.com>        http://kmself.home.netcom.com/
 What Part of "Gestalt" don't you understand?
    When everyone is out to get you, paranoia is only good thinking.


["application/pgp-signature" not shown]