Skip to content
 

NPR’s gonna NPR

I was gonna give this post the title, Stat Rage More Severe in the Presence of First-Class Journals, but then I thought I’d keep it simple.

Chapter 1. Background

OK, here’s what happened. A couple weeks ago someone pointed me to a low-quality paper that appeared in PPNAS (the prestigious Proceedings of the National Academy of Sciences), edited by the same person who approved the notorious himmicanes paper and who had earlier published a paper with one of the authors of the notoriously power pose paper. This new article was an attempt to understand the sources of “air rage” but unfortunately the actual analysis was a big uninterpretable multiple regression on some observational data.

Just to get the criticisms out of the way, here’s what I wrote, explaining why I couldn’t believe any of the claims in that article:

The interpretation of zillions of regression coefficients, each one controlling for all the others. For example, “As predicted, front boarding of planes predicted 2.18-times greater odds of an economy cabin incident than middle boarding (P = 0.005; model 2), an effect equivalent to an additional 5-h and 58-min flight delay (0.7772 front boarding/0.1305 delay hours).” What does it all mean? Who cares!

Story time: “We argue that exposure to both physical and situational inequality can result in antisocial behavior. . . . even temporary exposure to physical inequality—being literally placed in one’s “class” (economy class) for the duration of a flight—relates to antisocial behavior . . .”

A charming reference in the abstract to testing of predictions, even though no predictions were supplied before the data were analyzed.

They report a rate of incidents of 1.58 per thousand flights in economy seats on flights with first class, .14 per thousand flights in economy seats with no first class, and .31 per thousand flights in first class.

It seems like these numbers are per flight, not per passenger, but that can’t be right: lots more people are in economy class than in first class, and flights with first class seats tend to be in bigger planes than flights with no first class seats. This isn’t as bad as the himmicanes analysis but it displays a similar incoherence.

I didn’t explain all these points in detail—this was a blog post, not a textbook or even a referee report—but it was all there. To spell it out: My first quoted paragraph above addressed the problem of the regression coefficients, which is the same problem you noted in your comment. My second quoted paragraph is relevant in that the paper makes claims about human behavior which are not supported by their data. My third quoted paragraph pointed out the non-preregistered nature of the analysis. As always, preregistration is not required, but when this sort of completely open-ended study is not preregistered, this calls into question all claims of prediction accuracy and p-values. My fourth and fifth paragraphs address the point that the direct comparisons presented in the paper are uninterpretable.

Finally, the data are unavailable so it is impossible for an outsider to evaluate any of these claims. If the data were public, I’d recommend publication under a much lower standard, because once the data are out there, others could do their own analyses.

For another take on data problems with the “air rage” paper, see this post by John Walton.

Chapter 2. First mention of NPR

When posting on this study, I threw gratuitous shade at one of America’s most trusted news sources in my “tl;dr summary”:

NPR will love this paper. It directly targets their demographic of people who are rich enough to fly a lot but not rich enough to fly first class, and who think that inequality is the cause of the world’s ills.

The next day I posted a roundup of media outlets that’d fallen for this story, including CNN, the LA Times, and ABC News, along with respected tech sources Science and BoingBoing. I discussed the selection bias that occurs when the best science reporters realize this study is empty and don’t report it, while everyday journalists just follow the PPNAS label and don’t even think there could be a problem. All jokes about “stat rage” aside, this is a big problem in that consumers of the news only see the sucker takes, never the knowledge.

But NPR wasn’t included in that media roundup, hence I wrote “I was unfair to NPR,” and commenter Sepp asked,

Why the dig at NPR? And why the implication that NPR listeners cannot distinguish good scientific articles from bad ones that agree with listeners’ values? On that note, why the implicit indictment of said values (i.e. the desire to reduce inequality, etc.)? I find these statements saddening and confusing.

Chapter 3. The return of NPR

But then, after all that, NPR bit on the story—multiple times!

I’m sad to say that our public radio network lived up to its reputation. And I really am sad. I’d be much happier to report that they showed admirable skepticism and restraint. But they didn’t:

– Wait Wait Don’t Tell Me (or so I’ve been told; I haven’t seen the transcript so maybe they were actually mocking the study; I can only hope.)

Planet Money:

Screen Shot 2016-05-19 at 8.11.01 PM

– And finally, this from Alva Noë:

Screen Shot 2016-05-19 at 8.15.15 PM

The nation’s finest news source, indeed. It could’ve been worse: it could’ve been mentioned on NPR’s evening news show, but still, it’s disappointing.

Chapter 4. Summary

I return to my original statement:

NPR will love this paper. It directly targets their demographic of people who are rich enough to fly a lot but not rich enough to fly first class, and who think that inequality is the cause of the world’s ills.

NPR was not the only prestige outlet to be fooled.

Science magazine fell for this story (“Air rage? Blame the first-class cabin”). No surprise they were duped, I guess: tabloids gotta stick together.

But I was disappointed to see the usually-skeptical Economist take the bait too (“Resentment of first-class passengers can be a cause of air rage”). The Economist is no great enemy of inequality so they had no particular political reason to like this one. I guess they got conned by the PPNAS label.

Good job, PPNAS: another short-term win for the PPNAS publicity machine, another long-term loss for your reputation. Or, should I say, medium-term, as I sill have hope that you will clean up your act someday.

Let me conclude with this, from a commenter on the Economist article who understands this better than the Economist’s own reporter:

Screen Shot 2016-05-19 at 8.24.13 PM

I think NPR and these other news outlets can do better. And really, who goes into journalism to reheat press releases, anyway?

And, hey, to the authors of the paper: It’s ok. Everyone makes mistakes. Statistics is hard. I’m sure you were intending to do good science here. I’m not quite sure what to recommend for you. For this particular line of study: Sure, I’d recommend you cut your losses and release a short statement recognizing that the data don’t support your claims. After that? Maybe bring in a couple collaborators, one who knows about airlines and one who knows about causal inference. Sure, that takes work, but it’s kind of a necessity if (a) you want to learn anything useful from these data, and (b) you’re not willing or able to just make the data public. For future research on other topics: Hmmm, I guess my quick suggestion for any future paper is to present it at seminars in a few econ departments. Economists are pretty tough. They don’t catch every error but they might’ve caught these. And, for God’s sake, don’t submit to PPNAS anymore. Look what happened last time. In all sincerity, I wish you the best, and I urge you to reject the lure of the quick publication. PPNAS isn’t doing you any favors by publishing work like this (except, I guess, indirectly, in that now you’re getting some free advice from me). It’s a dysfunctional relationship we have here, between journals that seek publicity, news organizations that are all too willing to essentially run press releases, and researchers who often just don’t know better, and are led to believe that anything with “p less than .05” that’s published in a journal is good for them. Time to jump off the merry-go-round.

P.S. I wrote this post several months ago—this blog’s on a lag—and it just happened to appear today.

17 Comments

  1. Clyde Schechter says:

    “P.S. I wrote this post several months ago—this blog’s on a lag—and it just happened to appear today.”

    OK, but that’s true of nearly all of your posts. Is there some reason you made a point of mentioning it this time?

    • Andrew says:

      Clyde:

      I mentioned it because this post was about a PPNAS article that had been edited by Susan Fiske, and I didn’t want people to think that I was just posting for that reason. It’s just something that I wrote a few months ago.

  2. The study itself is an amusing read. It begins, “We posit that the modern airplane is a social microcosm of class-based society, and that the increasing incidence of “air rage” can be understood through the lens of inequality.” I would have taken this as a hoax; sadly, it isn’t.

    As for front boarding: Did they not consider the possibility that it might take longer than middle boarding, and that that might affect the patience of some of the boarders? (I don’t know that it does take longer–but it seems likely.)

    As for NPR, it’s no surprise–they’ve fallen for all sorts of nonsense, including the introvert/extravert stuff. Ironically, they have a piece today on flawed medical research: http://www.npr.org/sections/health-shots/2016/09/29/495797739/flawed-research-tool-leads-to-faulty-medical-findings

    On a tangent, I hope someone reveals the flaws in Cain’s frequent assertions that introverts and extraverts have “literally, different nervous systems.” She told NPR the following in an interview (http://www.npr.org/sections/ed/2016/02/18/465999756/how-parents-and-teachers-can-nurture-the-quiet-power-of-introverts):

    “It’s really not different for a child than for an adult. It’s a person who feels at their best and at their most alive when they’re in quieter, more mellow environments. And it stems from a neurobiological difference between introverts and extroverts. Literally, different nervous systems. Introverts have nervous systems that simply react more to everything that’s going on around them, and that means they feel more in their sweet spot when there’s less stuff happening. And extroverts have nervous systems that react less, which means that they don’t get to their sweet spot until there’s more stuff happening.”

    I have no problem with someone articulating his or her own ideas about introversion and extraversion. It’s potentially a rich subject. But if you invoke science, please don’t make such a mess of it. It’s embarrassing. And NPR, wake up.

    • Alex Gamma says:

      Note the explanatory vacuity of invoking a neural difference in this case. If you’re a physicalist, you already assume that every psychological difference must be underwritten by a neural difference. So no news here. If you’re not a physicalist, then it is not obvious that whatever neural difference there might be would matter.

      In my experience, mentioning that some brain correlate of mental state/concept X has been found is often what makes X real for people. It’s like merely saying that the brains of people with and without X are different counts as an explanation for X, and the question why the brains are different in the first place does not even have to be asked.

      • True, but there’s also a question of definition. Who are these “introverts” and “extraverts”? Cain claims repeatedly that introverts make up one-third to one-half of the population. Yet research studies often focus on a fraction of that group: those who score especially high or low on introversion/extraversion tests.

        Thus many research findings (incomplete and tentative ones at that) are presented as though they apply to a population they do not represent in the first place.

  3. Jeff Lemons says:

    to be fair to journalists, even if they’re nonsense, the papers went through peer review. So the journalist effectively has 3 peer reviewers, a set of authors, and an editor telling them that this is legit.

  4. Rahul says:

    “Academics are gonna do what Academics do”

  5. Sean S says:

    You can’t do “Junk Science” without “Junk Stats.”

    I’m not worried since this will all be fixed with Machine Learning. ;)

  6. Larry says:

    On today’s NPR morning show, Scott Simon?? had a segment on the power pose research and had Uri Simonsohn on talking about the problems with it.

Leave a Reply