I noticed this on Retraction Watch:
“Scientists clearly cannot rely on the traditional avenues for correcting problems in the literature.” PubPeer responds to an editorial slamming the site.
I’ve never actually read anything on PubPeer but I understand it’s a post-publication review site, and I like post-publication review.
So I’m heading into this one on the side of PubPeer, and let me deflate any suspense right here but telling you that, having followed the links and read the discussion, my position hasn’t changed.
So, no news and no expectation that this new story should change your beliefs, if you happen to be on the Evilicious side of this particular debate.
So, if I’m not trying to convince anybody, why am my writing this post? Actually, I’m usually not trying to convince anyone when I write; rather, I use writing as a way to explore my thoughts and to integrate the discordant information I see into coherent stories (with one sort of coherent story being of the form, “I don’t yet understand what’s going on, the evidence seems to be contradictory, and I can’t form a coherent story”).
In that sense, writing is a form of posterior predictive check, or perhaps I should just say posterior inference, a way of working out the implications of my implicit models of the world in the context of available data.
They say Code Never Lies and they’re right, but writing has its own logic that can be helpful to follow.
Hence, I blog.
Now back to the item at hand. The above link goes to a post on PubPeer that begins as follows:
In an editorial entitled “Vigilante science”, the editor-in-chief of Plant Physiology, Michael Blatt, makes the hyperbolic claim that anonymous post-publication peer review by the PubPeer community represents the most serious threat to the scientific process today.
We obviously disagree. We believe a greater problem, which PubPeer can help to address, is the flood of low-quality, overinterpreted and ultimately unreliable research being experienced in many scientific fields . . .
I then clicked to see what Michael Blatt had to say in the journal Plant Physiology.
Since its launch in October 2012, PubPeer has sought to facilitate community-wide, postpublication critique of scientific articles. The Web site has also attracted much controversy . . . .
PubPeer operates as a blog on which anyone can post comments, either to a published article or to comments posted by other participants, and authors may respond. It is a bit like an extended journal club; not a bad idea to promote communication among scientists, you might think, so why the controversy?
Why, indeed? Blatt explains:
The problems arising are twofold . . . First, most individuals posting on PubPeer—let’s use the euphemism commenters for now—take advantage of the anonymity afforded by the site in full knowledge that their posts will be available to the public at large.
I don’t understand why “commenters” is considered a euphemism. That’s the problem with entering a debate in the middle—sometimes you can’t figure out what people are talking about.
Second, the vast majority of comments that are posted focus on image data (gels, blots, and micrographs) that contribute to the development of scientific ideas but are not ideas in themselves. With few exceptions, commenters on PubPeer do no more than flag perceived faults and query the associated content.
But, wait, what’s wrong with commenting on image data? And “flagging perceived faults”—that’s really important, no? We should all be aware of faults in published papers.
Of course, I say this as someone who’s published a paper that was invalidated by a data error, so I personally would benefit from outsiders checking my work and letting me know when they see something fishy.
So what’s the problem, then? Blatt tells us:
My overriding concern with PubPeer is the lack of transparency that arises from concealing the identities of both commenters and moderators.
This is so wrong I hardly know where to start. No, actually, I do know where to start, which is to point out that articles are published based on anonymous peer review.
Who were the reviewers who made the mistake of recommending publication of those papers by Daryl Bem or Satoshi Kanazawa or those ovulation-and-voting people? We’ll never know. For the himmicanes and hurricanes people, we do know that Susan Fiske was the editor who recommended publication, and she can be rightly criticized for her poor judgment on this one (nothing personal, I make lots of poor judgments myself, feel free to call me out on them), but we don’t know who were the external referees who failed to set her straight. Or, to go back 20 years, we don’t know who were the statistical referees who made the foolish, foolish decision to recommend that Statistical Science publish that horrible Bible Code paper. I do know the journal’s editor at the time, but he was in a difficult position if he was faced with positive referee reports.
So, according to Blatt: Anonymous pre-publication review, good. Anonymous post-publication review, bad. Got it.
Indeed, Blatt is insistent on this point:
I accept that there is a case for anonymity as part of the peer-review process. However, the argument for anonymity in postpublication discussion fallaciously equates such discussion with prepublication peer review. . . . In short, anonymity makes sense when reviews are offered in confidence to be assessed and moderated by an editor, someone whose identity is known and who takes responsibility for the decision informed by the reviews. Obviously, this same situation does not apply postpublication, not when the commenters enter into a discussion anonymously and the moderators are also unknown.
Oh no god no no no no no. Here’s the difference between pre-publication reviews, as usually conducted, and post-publication reviews:
Pre-publication reviews are secret. Not just the author of the review, also the actual content. Only very rarely are pre-publication reviews published in any form. Post-publication reviews, by their very nature, are public.
As Stephen King says, it’s the tale, not he who tells it. Post-publication reviews don’t need to be signed; we actually have the damn review. Given the review, the identity of the reviewer supplies very little information.
The other difference is that pre-publication reviews tend to be much more negative than post-publication reviews. I find it laughable when Blatt writes that post-publication reviews are “one-sided,” “petty,” “missing . . . courtesy and common sense,” “negative and occasionally malicious,” and “about policing, not discussion.” All these descriptions apply even more for pre-publication reviews.
Why do I care?
At this point, you might be asking yourself why I post this at all. Neither you nor I have ever heard of the journal Plant Physiology before, and we’ll likely never hear of it again. So who cares that the editor of an obscure journal emits a last-gasp rant against PubPeer, a site with represents the future in the same way that editor-as-gatekeeper Michael Blatt represents the past.
Who indeed? I don’t care what the editor of Plant Physiology thinks about post-publication review. What I do care about is we’re not there yet. Any dramatic claim with “p less than .05” that appears in Science or Nature or PPNAS or Psychological Science still has a shot of getting massive publicity. That himmicanes-and-hurricanes study was just last year. And this year we’ve seen a few more.
P.S. Incidentally, it seems that journals vary greatly in the power they afford to their editors. I can’t imagine the editor of Biometrics or the Journal of the American Statistical Association being able to publish this sort of opinion piece in the journal like this. I don’t know the general pattern here, but I have the vague impression that biomedical journals feature more editorializing, compared to journals in the physical and social sciences.
P.P.S. Two commenters pointed out small mistakes in this post, which I’ve fixed. Another point in favor of post-publication review!