Alex Braunstein writes about Klout, a company which measures Twitter/Facebook influence:
As a Ph D statistician and search quality engineer, I [Braunstein] know a lot about how to properly measure things. In the past few months I’ve become an active Twitter user and very interested in measuring the influence of individuals. Klout provides a way to measure influence on Twitter using a score also called Klout. The range is 0 to 100. Light users score below 20, regular users around 30, and celebrities start around 75. Naturally, I was intrigued by the Klout measurement, but a careful analysis led to some serious issues with the score. . . .
Braunstein continues with some comparisons of different twitter-users and how their Klout scores don’t make much sense. I don’t really see the point of the Klout scores in the first place: I guess they’re supposed to be a quick measure to use in pricing advertising? Whatever, I don’t really care.
What did interest me was a remark on Braunstein’s blog:
Everything in life can be measured. Some quantities live on natural measurement scales: height, weight, temperature, etc. Some quantities are derived measurements: happiness, deliciousness, hunger, etc. Though all useful measurements, research has repeatedly shown derived measurements to be inconsistent and not trustworthy individually. Specifically, if two individuals tell you their happiness levels are an 8 and a 9 on a scale of 10, we have no way to know: what this means for each individual without significant amounts of context, and which individual is “happier” even if 8 is less than 9.
I think Braunstein is on to something but I would frame it slightly differently. Happiness is typically defined as responses in a survey interview. So, no, I wouldn’t want to call it a derived quantity. I’d rather call it a subjective measurement.
The problem with the Klout score is not that it’s subjective but that it’s cloudy: we don’t know what it is. To understand a cloudy measurement, one has to poke it from the outside. With Google, people have done this using google fights, google trends, etc., and they find interesting things. Braunstein tries this with Klout and finds confusion.
Which makes sense given that Klout itself seems like a tool for . . . selling itself! Sort of other notorious rating schemes such as the Places Rated Almanac and the U.S. News college ratings.
Let’s try a baseball analogy. Hits, walks, home runs, stolen bases, even goofy statistics such as “saves”–all of these are direct measurements or nearly so (accepting that we still have to handle decisions about errors, sacrifices, etc.). Batting average, on-base percentage, earned run average, etc.: these are derived quantities. Some more esoteric derived quantities such as Runs Produced will be accepted only if they offer some benefits in understanding.
P.S. More from John Scalzi. If only he’d read this blog, he would’ve known about this ten months ago!