We’ve had lots of lively discussions of fatally-flawed papers that have been published in top, top journals such as the American Economic Review or the Journal of Personality and Social Psychology or the American Sociological Review or the tabloids. And we also know about mistakes that make their way into mid-ranking outlets such as the Journal of Theoretical Biology.
But what about results that appear in the lower tier of legitimate journals? I was thinking about this after reading a post by Dan Kahan slamming a paper that recently appeared in PLOS-One. I won’t discuss the paper itself here because that’s not my point. Rather, I had some thoughts regarding Kahan’s annoyance that a paper with fatal errors was published at all. I commented as follows:
Read between the lines. The paper originally was released in 2009 and was published in 2013 in PLOS-One, which is one step above appearing on Arxiv. PLOS-One publishes some good things (so does Arxiv) but it’s the place people place papers that can’t be placed. We can deduce that the paper was rejected by Science, Nature, various other biology journals, and maybe some political science journals as well.
I’m not saying you shouldn’t criticize the paper in question, but you can’t really demand better from a paper published in a bottom-feeder journal.
Again, just because something’s in a crap journal, doesn’t mean it’s crap; I’ve published lots of papers in unselective, low-prestige outlets. But it’s certainly no surprise if a paper published in a low-grade journal happens to be crap. They publish the things nobody else will touch.
It does make you wonder, though, what can be believed at all! Reading the title and even the abstract is not enough.
P.S. The discussion in the comments section is interesting but on a different topic than I was writing about. I think it’s safe to assume that the paper being discussed above was rejected by the tabloids before appearing in PLOS-One. That’s not a surprise: People submit a lot of weak papers to the tabloids and most of them get rejected. When a weak paper appears in a tabloid, lots of people (including me) get annoyed, and we say things like, How could Science (or Nature or Psych Science) publish that crap! Even non-tabloid serious journals publish seriously flawed papers; in recent months we’ve discussed examples from the American Economic Review and the American Sociological Review. Again, in these cases we can get annoyed that major errors slipped past the review process.
But when a weak paper appears in PLOS-One, we cannot be shocked. PLOS-One by design will regularly publish weak papers along with the good stuff.
Many commenters remarked on the high quality of much of what appears in PLOS-One and urged us all not to restrict our reading to the so-called top journals. I agree completely. Many, maybe most, of my most influential papers over the years have not appear in top journals, but they’re still good papers and represent important research!
But that’s my point. When we encounter crap in leading journals, we know how to react—with annoyance. To the extent we go beyond the top journals, we will still find a lot of crap even when not looking for it (with an extreme case being Dan Kahan’s encounter with the paper described above), and we need a way of thinking about it. Annoyance is not appropriate, and certainly it’s not appropriate to believe the claims in the abstract of a paper, just because it’s published.
To say it again: the point of this post is not to bash PLOS-One or the Journal of Theoretical Biology or other low-ranked but legitimate journals, it’s to explore how we should think about the papers that appear in such places. The traditional approach is to assume that everything published in a peer-reviewed journal is correct, but that’s not right.