A few months ago Psych Science issued a press release, “Blue and Seeing Blue: Sadness May Impair Color Perception,” promoting a fatally flawed paper that appeared in their journal. I heard about this paper from Nick Brown, and we slammed it on the blog.
As I wrote at the time, I have nothing against the authors of the paper in question. I expect they’re doing their best. It’s not their fault that (a) statistical methods are what they are, (b) statistical training is what is is, and (c) the editors of Psychological Science don’t know any better. It’s all too bad, but it’s not their fault. I laugh at these studies because I’m too exhausted to cry, that’s all. And, before you feel too sorry for these guys or for the editors of Psychological Science or think I’m picking on them, remember: if they didn’t want the attention, they didn’t need to publish this work in the highest-profile journal of their field. If you put your ideas out there, you have to expect (ideally, hope) that people will point out what you did wrong.
I’m honestly surprised that Psychological Science is still publishing this sort of thing. They’re really living up to their rep, and not in a good way. PPNAS I can expect will publish just about anything, as it’s not peer-reviewed in the usual way. But Psych Science is supposed to be a real journal, and I’d expect, or at least hope, better from them.
The good news and the bad news
The good news comes from a commenter, who reports that Psych Science just retracted the paper:
The authors still express what I view as naive and unrealistic hopes:
We will conduct a revised Experiment 2 that more directly tests the motivational interpretation and improves the assessment of BY accuracy. If this revised experiment yields the same findings as our original Experiment 2, we will seek publication of our original Experiment 1 with the new Experiment 2. We remain confident in the proposition that sadness impairs color perception, but would like to acquire clearer evidence before making this conclusion in a journal the caliber of Psychological Science.
I think they don’t fully understand how difficult it is to learn from noisy data. But I’m glad they retracted. And I can hardly blame them for still holding out hope in their hypothesis.
The bad news is that Psych Science has not yet promoted the retraction at the same level as it promoted the original claim.
What I’d like to see from them is a feature story, titled something like, “Blue and Seeing Blue: Desire for Publication May Impair Research Effectiveness.” Instead, though, this is what I see on their webpage:
They did tweet the retraction, so that’s something:
And they retracted the old press release. But I really think they should give the retraction the same publicity they gave to the original report.
Again, no shame on the researchers involved. They made a mistake, something that happens all the time as is no surprise given the null hypothesis significance testing approach in which researchers are trained. I make statistical mistakes all the time, so I’m not surprised that others do too. Post-publication peer review is a great way to catch such errors, and increased awareness of the problems with noisy studies may be a way to reduce such errors in the future.
P.S. More on the story at Retraction Watch.
While I agree that Psych Science could do more to publicize the retraction and that the authors perhaps do not understand the limitations of their empirical studies, the fact that the article is retracted is a _big_ deal. It suggests that people in the field actually _care_ about getting things right. Given author reactions for similar types of criticism of other papers, there were reasons to doubt that was the case. Kudos to the authors, editor, and critics for doing the right thing.
The interim editor at Psych Science just published an editorial about replication and statistics.
http://pss.sagepub.com/content/early/2015/11/09/0956797615616374.full.pdf+html
Readers of this blog may disagree with some of the details in the editorial, but it seems to me that the editor is making a sincere effort to improve the quality of what gets reported in Psych Science.
Is there a general recommendation for when authors should request a retraction and when they should just publish a correction (e.g. like the correction Gelman and Cai published for the “Should the Democrats Move to the Left on Democratic Policy” paper)?
If we retract all papers where errors have been identified, there might be nothing left in the libraries :) Also, is it not better to have the papers with the errors available to everyone so that we can learn from the error?
Could the authors of the sadness and colour perception paper just issue a correction like “..all the analyses in this paper are fundamentally flawed … and everything should be assumed to be wrong …”? If that statement was made public and prominent, then no one would be misled by the paper and the fact that is fundamentally flawed would be made completely clear but yet we still have the paper on record to learn from it, even if it is just as a cautionary tale.
Mark:
Yes, I agree that a correction would be fine. I don’t really see much distinction between retraction and correction. In either case, the point is that part of the paper as published was in error.
In the public consciousness, or at least in mine, a retraction is often an admission of something dishonourable or fraudulent. I don’t think it should necessarily be seen like this, but I think it does have this connotation. (Just looking at the stories on retraction watch, we see how a retraction is sometimes the act of final capitulation and admission of fraud by the authors or perhaps a prelude to some perhaps serious sanction by an institution or funder.)
I think many people would be loath to admit their errors if doing so required them to retract the paper just because a retraction has such a negative stigma attached to it.
A published correction, on the other hand, even one that is essentially saying that everything in the paper is wrong and nothing should be taken as reliable, has no such dishonourable or career damaging connotation. Everyone makes mistakes and sometimes those mistakes entail that the whole study is irredeemably flawed, but admitting this is no reason for shame or disgrace, and so people may more willing to do so than to retract the paper.
I also think that a correction has the advantage of keeping everything on archival record. We would have the original paper and could learn from it (if only to learn from its mistakes), and the correction itself could contain valuable discussion about the nature of the errors and how they arose, and such a discussion would require the original paper too.
I disagree that retractions imply fraud or deceit. There are certainly papers that are retracted because their conclusions are realized to be wrong, and I think everyone knows this. It is also not the case that retracted papers disappear; as one example, see http://www.sciencemag.org/content/347/6218/139.1.short , and click PDF to the left. (The fact that it is retracted, by the authors, due to error of interpretation is clearly stated.) I could have linked my (one) retracted paper, but I will admit that this is painful to do! The “archival record” still exists.
“Corrections,” on the other hand, imply that there are really minor errors, perhaps in a figure, or some subset of the data, that don’t affect the overall conclusion. In practice, if I see a correction, I think: ok, if I’m going to do something that builds on this work, I should read the correction to make use of it, and if I’m not there’s no need to read it. If I see a retraction, I think: that entire conclusion is flawed (for whatever reason).
Raghuveer:
I think the terms “correction” and “retraction” have varying meanings based on context, and I’m wary of an attempt to draw a hard distinction in this way. I really think of the two as the same thing. But perhaps in psychology the terms are defined less ambiguously. In any case, it would be fine for me if journals issued lots and lots of corrections. If retractions bother people, I’d be happy to just talk about corrections.
Good points.
I partly agree, but I think there needs to be some room for a full retraction (e.g. for fraud). The problem is that neither retractions nor corrections are handled well by the present systems.
Hopefully the original reviewers and the action-editor for this paper are getting some pointed questions from the editor. This was the type of error that about 50% of my first-year grad stats class would catch.
At least Psych Science is doing the right thing by retracting – more than can be said for that “authentic leadership” paper that you discussed some time ago and that completely hinges on obviously wrong (i.e., impossible) findings.
http://statmodeling.stat.columbia.edu/2015/07/11/inauthentic-leadership-development-and-validation-of-methods-based-criticism/
Did you ever get a chance to ask your own action-editor from Journal of Management about that paper?
While the statistics might have been wrong in this paper, I do not think the hypothesis itself is as naive as is being argued here. Think about the fact that we often describe colors in terms of their warmth, which is an emotion ladened word. There may even be a literature on this. If the same part of the brain that processes this experience of warmth also processes sadness, then an effect on color perception is indeed plausible. Though the fact that it is blue and we experience the blues may be a little too serendipitous for comfort, it is nonetheless does not seem a bizarre hypothesis.
James:
I agree that the hypothesis is not bizarre. But I don’t see this study, even if analyzed correctly, as adding anything to our understanding of the topic. If the authors want to write a theoretical paper on the subject, that’s fine, and it can be judged as such. My problem is with papers of this sort where the theory is vague and the data are weak, but publication ensues because the theory is taken to support the data analysis, and the data analysis is taken to support the theory.
Point taken.
The source of all this is how we let people get away with what they call training and still get degrees.
Luca:
It’s not so simple, given that many or most of the teachers and the textbooks emphasize null hypothesis significance testing. Not to mention the incentive structure by which purported discoveries get published in top journals and promoted by APS, BPS, etc.
I agree that this is extremely hard to fix. Maybe a good step to the right direction would be to emphasise more formal epistemology. Requiring aspiring scientist to have a relatively coherent account about why they think their methods have anything to do with what’s true and what’s not would help a little, and I think a lot of students would realise that their teachers don’t have this – so maybe they should look for others. You may of course say that I put too much hope in the uncorrupted youth. But it would also be harder to get away with no mathematical skills if formal logic is routinely on the table, so the youth will also have better clues whether they should pursue a research career or not, so as a population, they would have better scientific intuition than now.
The incentive structure is a largely independent problem in my opinion, and I share your pessimism in that department.
I’m a bit disturbed by their plans to repeat their experiment and publish if they get the result they want (and presumably repeat as necessary). Some lessons not being learned there I think.
Granted the publication system is set up to publish false positives, but I do not understand the argument that a poorly done study should not be replicated. Though if it involves inducing depression, ala facebook manipulations, then you would have a strong ethical argument against it.