Bill Jefferys points me to this news article and writes:
Looks at first glance like another NPR example of poor statistics, but who knows?
I took a look and here were my thoughts, in order of occurrence:
NPR . . . PPNAS . . .
Also this: “the results were consistent across all the societies studied.” There’s no way they got positive findings for all 24 societies.
OK, let’s look at the paper.
Edited by Susan “Himmicane” Fiske . . . uh oh . . .
On the other hand, the claim is not implausible.
Let’s look at the key figure:
Hey—for the FF dyads they really do identify pairs of friends more accurately than chance for all 24 countries. How could that be???
Wait—I get it! For each dyad, participants had to guess Friend or Stranger. So what happened? Pairs of women were consistently guessed as Friends.
New headline: In 24 Countries, People Think Women Are More Friendly Than Men.
So, unless I”m missing something, the researchers made a classic statistical error: they evaluated a prediction by conditioning on the observed value. You can’t do that: you have to condition on the prediction.
Am I right here? Let’s check these intuitions by focusing on the U.S. and approximately reading the numbers off the graph. I divide each histogram bar height by 2, under the assumption that 50% of the true values are strangers and 50% are friends in each category.
Male and mixed dyads:
28% of the time it was a pair of strangers correctly identified as strangers
22% of the time it was a pair of strangers wrongly identified as friends
17% of the time it was a pair of friends wrongly identified as strangers
33% of the time it was a pair of friends correctly identified as friends
So, if you say Friend, there’s a 60% chance the person was actually a friend. If you say Stranger, there’s a 62% chance the person was actually a stranger. Pretty good!
16% of the time it was a pair of strangers correctly identified as strangers
34% of the time it was a pair of strangers wrongly identified as friends
2% of the time it was a pair of friends wrongly identified as strangers
48% of the time it was a pair of friends correctly identified as friends
So, if you say Friend, there’s a 59% chance the person was actually a friend. If you say Stranger, there’s an 89% chance the person was actually a stranger. Pretty good too!
OK, so now I’m starting to think they’re on to something . . .
Yes there seem to be flaws in the analysis but, based on the data, it doesn’t seem to be just noise mining.
In particular, the statement from the paper, “Across all societies, listeners were able to distinguish pairs of colaughers who were friends from those who were strangers that had just met.”
Is this a big deal? I have no idea. The one thing I couldn’t figure out is how did they get the “brief, decontextualized instances of colaughter” from their recordings. If people are talking for awhile there could me more than one instance of laughing. Maybe strangers talking won’t laugh at all? I looked at the paper and the supplementary material but could find no instances where it said how they extracted the laughs. It’e probably there but I didn’t notice it. So I’ll reserve judgment on this one until I find out how the data were put together.
In any case, I wanted to share this, just so all of you can see my expectations confounded.