To busy readers: Skip to the tl;dr summary at the end of this post.
A psychology researcher sent me an email with subject line, “There’s a hell of a paper coming out in PPNAS today.” He sent me a copy of the paper, “Physical and situational inequality on airplanes predicts air rage,” by Katherine DeCelles and Michael Norton, edited by Susan Fiske, and it did not disappoint. By which I mean it exhibited the mix of forking paths and open-ended storytelling characteristic of these sorts of PPNAS or Psychological Science papers on himmicanes, power pose, ovulation and clothing, and all the rest.
There’s so much to love (by which I mean, hate) here, I hardly know where to start.
– Coefficient estimate and standard errors such as “1.0031** (0.0014)” (yes, that’s statistically significantly different from the baseline value of 1.0000).
– Another coefficient of “11.8594” (dig that precision) with a standard error of “11.8367” which is still declared statistically significant at the 5% level. Whoops!
– The ridiculous hyper-precision of “Flights with first class present are ∼46.1% of the population of flights” (good thing they assured us that it wasn’t exactly 46.1%).
– The interpretation of zillions of regression coefficients, each one controlling for all the others. For example, “As predicted, front boarding of planes predicted 2.18-times greater odds of an economy cabin incident than middle boarding (P = 0.005; model 2), an effect equivalent to an additional 5-h and 58-min flight delay (0.7772 front boarding/0.1305 delay hours).” What does it all mean? Who cares!
– No raw data. Sorry, proprietary restrictions so nobody can reproduce this analysis! (Don’t get me wrong, I have no problem with researchers learning from proprietary information, I do it all the time. What the National Academy of Sciences is doing publishing this sort of thing, I have no idea. Or, yes, I do have an idea, but I don’t like it.)
– Story time: “We argue that exposure to both physical and situational inequality can result in antisocial behavior. . . . even temporary exposure to physical inequality—being literally placed in one’s “class” (economy class) for the duration of a flight—relates to antisocial behavior . . .”
– A charming reference in the abstract to testing of predictions, even though no predictions were supplied before the data were analyzed.
The authors don’t share any of their data, but they do say that there were between 1500 and 4500 incidents in their database, out of between 1 and 5 million flights. So that’s about 1 incident per thousand flights.
They report a rate of incidents of 1.58 per thousand flights in economy seats on flights with first class, .14 per thousand flights in economy seats with no first class, and .31 per thousand flights in first class.
It seems like these numbers are per flight, not per passenger, but that can’t be right: lots more people are in economy class than in first class, and flights with first class seats tend to be in bigger planes than flights with no first class seats. This isn’t as bad as the himmicanes analysis but it displays a similar incoherence.
There’s no reason we should take this sort of tea-leaf-reading exercise seriously. Or, to put it another way—and I’m talking to you, journalists—just pretend this was published in some obscure outlet such as the Journal of Airline Safety. Subtract the hype, subtract the claims of general relevance, just treat it as data (which we don’t get to see).
I should perhaps clarify that I can only assume these researchers were trying their best. They were playing by the rules. Not their fault that the rules were wrong. Statistics is hard, like basketball or knitting. As I wrote a few months ago, I think we have to accept statistical incompetence not as an aberration but as the norm. Doing poor statistical analysis doesn’t make Katherine DeCelles and Michael Norton bad people, any more than I’m a bad person just cos I can’t sink a layup.
NPR will love this paper. It directly targets their demographic of people who are rich enough to fly a lot but not rich enough to fly first class, and who think that inequality is the cause of the world’s ills.
P.S. I was unfair to NPR. See here.