Skip to content
 

I’m really getting tired of this sort of thing, and I don’t feel like scheduling it for September, so I’ll post it at 1 in the morning


A couple days ago I received an email:

I’m a reporter for *** [newspaper], currently looking into a fun article about a recent study, and my old professor *** recommended I get in touch with you to see if you would give me a comment on the statistics in the study.

It’s a bit of fun, really, but I want to approach it with real scientific analysis, so your voice in the piece would be great.

The study is here: http://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0153419#sec010

I was eager to avoid doing any real work, so I read the paper and sent the journalist a reply:

The paper is cute but I think their conclusions go beyond their data. I have two concerns:

1. It’s not clear to what extent should we be willing draw general conclusions from Mechanical Turk participants to voters in general.

2. The difference between “significant” and “not significant” is not itself statistically significant (see here: http://www.stat.columbia.edu/~gelman/research/published/signif4.pdf)
This is relevant in considering all their comparisons between candidates. For example, they write, “the results revealed that holding favorable views of three potential Republican candidates for US president (Ted Cruz, Marco Rubio, and Donald Trump) was positively related to judging bullshit statements as profound. In contrast, non-significant relations were observed for the three Democratic candidates (Hillary Clinton, Martin O’Malley, and Bernie Sanders).” But these differences could be explained by noise.

3. Multiple comparisons. The data are open-ended. They compared Democrats to Republicans but they could’ve compared male to female candidates, and they also could’ve looked at age and sex of respondents. So it’s not a surprise that if they looked through these data, they could find statistically significant comparisons (see here: http://www.stat.columbia.edu/~gelman/research/published/ForkingPaths.pdf)

That was all, but then I just received an email from Dan Kahan pointing to a long post by Asheley Landrum attacking the study on various grounds, including the points above.

I’m assuming the study was a parody (consider the paper’s title!) so I think Landrum may be going a bit over the top in the criticisms. Anyway, since this was all out there I thought I’d share my quick take above.

For the general issue of how reporters can write about this sort of science paper, see the templates here.

P.S. The good news is that no news organizations seem to have picked up this story. I’d like to think the news media has learned something after being embarrassed by their credulous coverage of the air rage story, but I suspect the big difference is that this new study was published in lowly PLOS-One, not the mighty PPNAS. The one journalist who contacted me seems to have made the wise decision to not write about this paper.

11 Comments

  1. Ibn says:

    I really wonder whether an opposite claim would have gotten past the editors. Of course, there is no way to know.

  2. BenK says:

    I hear you, but I don’t think it was a parody. There is an entire subfield that I discovered from their references.
    Anyway, those references didn’t make me feel particularly better about the foundations of the subject. *sigh*

  3. Paul Shearer says:

    Point (2) is well-taken but, just playing devil’s advocate here: if candidate favorability ratings X_j are uncorrelated with ratings of bullshit statements Y_j for j = 1 through N (number of candidates), then the sign of the sample correlation should be an unbiased coin toss right? So the probability of the result they got is 1/2^6 = 0.0156, which would pass significance if they had “mentally pre-registered” their partisan hypothesis :)

  4. Paul Shearer says:

    Oops, I forgot, they didn’t actually report the signs of the correlations, only the significance. Strike the previous comment…

  5. dmk38 says:

    this is just run of the mill/mine case of NHT disease. Authors reject null at p <0.05 & declare victory for claim about how world works that no one who looks at data & *thinks* about what inferences they support would accept (that's what Landum's scatterplots & lowess lines show). Neither a better sample nor another significance test (difference between BS scores of all D-candidate vs. all R-candidate supporters) would come close to a cure; the only remedy is to excise the malignant NHT craft norm from the body of social science

  6. Nearly 90,000 views and over 6,000 shares in just a week since publication–talk about high visibility publication, though I guess it’d be even higher if the story had been featured on NPR or some such.

    Interestingly, the PLoS editor posted the following comment on the article in response to some criticism from readers:

    “This article reports scientific research. It has what science requires. It is related to previous research, both on BS but also on more general topics (e.g., conservative tendencies, the impact of favorable views), and the possibility that “the more individuals have favorable views of persons talked about as potential Republican (conservative) candidates for US president the more they see profoundness in BS statements” is in fact a testable idea. Key measures were reliable (at least in terms of internal consistency, or alpha), and Spearman’s rho was a good choice, given the skew and sample size. All of the above was examined via the PLOS ONE peer review process. I see evidence for the conclusions offered by the authors, as you would expect of social science.”

  7. curio says:

    The attention paid to that “research” seems to suggest that “misperceiving bullshit as profound” characterizes a large part of the progressive crowd

  8. neutral observer says:

    I’m wondering how many people actually read the paper. I was curious, so I took a look at the strength of their conclusions based on the data. This is in their discussion (it is the majority of their discussion, in fact):

    “In the following paragraphs, we emphasize what can and what cannot be deduced from the study. First of all, we found small-to-medium sized correlations between holding favorable views of Republican candidates, conservatism, and bullshit receptivity. Given these effect sizes, it must be noted that far from all conservatives see profoundness in pseudo-profound bullshit statements. Certainly, some liberals also see profoundness in bullshit statements, while some conservatives may clearly reject profoundness in bullshit statements. Nevertheless, there is an overall tendency for conservatives relative to liberals to see profoundness in bullshit statements.

    Second, we want to note that the sample of the present study probably is not representative of the US as our study is restricted to the specific sample of Amazon Mechanical Turk workers and has a relatively small sample size for an online survey. Thus, one cannot make inferences about the entire population of the US (or other populations of other countries). However, this does not undermine the significance of the present research. As is usual in psychological studies, associations between constructs are tested. Whether these associations hold for different populations is surely interesting, but beyond the scope of most psychological studies, including the present research. In this context, we also want to note that some participants might be non-naïve or trustworthy, as some participants regularly and systematically participate in online studies via the Amazon Mechanical Turk platform [12]. Nonetheless, research shows that valid results in psychological research can be obtained using Amazon Mechanical Turk [10,13].

    Third, we want to emphasize that the present study is correlational in nature. Thus, no causal inferences can be drawn. One cannot conclude that conservatism leads to bullshit receptivity. What can be concluded is that conservatism is positively associated with seeing profoundness in bullshit statements and that those who have favorable views of Donald Trump, Ted Cruz, and Marco Rubio are more likely to judge bullshit statements as profound compared with individuals who have less favorable views of these candidates.

    Fourth, the present research remains empirically silent regarding the explanatory variable of why conservatism is positively associated with seeing profoundness in bullshit statements. We base our study on the empirically supported assumption found in other research that conservatives (vs. liberals) are less likely to use a reflective and critical thinking mode [8,9], a mode that is necessary to detect the vacuity of pseudo-profound bullshit statements [1]. However, reflective and critical thinking is not measured in the present study. The assumption that reflective and critical thinking can explain the found effects needs to be tested in future research. Additionally, as bullshit receptivity is related to religiousness [1], and while it is likely that religiousness is positively related to favorable views of Cruz, it could be that religiousness might explain (part of) the found relations. Since religiousness is not assessed in the present study (which also holds true for other demographic variables such as social class, income, and education) there is room for future research to test variables that might explain the found relations.

    Fifth, it is likely that conservatives are specifically receptive to pseudo-profound bullshit but not to others forms of bullshit. To detect pseudo-profound bullshit, a critical thinking mode is necessary, a mode that is less likely to be found in conservatives [5,8,9]. Another thinking mode may be required to detect other forms of bullshit, for instance, when people tell sincere exaggerations that are detectable without critical thinking [14]. As such, it seems possible that conservatism is not related to seeing any form of bullshit as profound.”

    This is not what I expected given the (apparently flippant) rejection of this paper.

    • Andrew says:

      Neutral:

      It is good that the authors of the paper do not hype its conclusions. But I don’t think the conclusions are supported by the data. So I think it’s a low-quality research paper, irrespective of it not being hyped. Were there hype, I’d like the paper even less. And, as I’ve said before in the contexts of other bad papers, I have no animosity toward the authors, I just don’t think they have a good understanding of statistical variation. That’s the way it goes. Statistics is hard.

Leave a Reply