I recently became aware of two new entries in the ever-popular genre of, Our Peer-Review System is in Trouble; How Can We Fix It?
Political scientist Brendan Nyhan, commenting on experimental and empirical sciences more generally, focuses on the selection problem that positive rather then negative findings tend to get published, leading via the statistical significance filter to an overestimation of effect sizes. Nyhan recommends that data-collection protocols be published ahead of time, with the commitment to publish the eventual results:
In the case of experimental data, a better practice would be for journals to accept articles before the study was conducted. The article should be written up to the point of the results section, which would then be populated using a pre-specified analysis plan submitted by the author. The journal would then allow for post-hoc analysis and interpretation by the author that would be labeled as such and distinguished from the previously submitted material. By offering such an option, journals would create a positive incentive for preregistration that would avoid file drawer bias. More published articles would have null findings (at least 5%!), but that’s how science is supposed to work.
Quality control could be maintained by “replication audits of a random subset of published articles. At a minimum, these audits would verify that all the results in an article could be replicated. They could conceivably go further in some cases and try to recreate the author’s data and results from publicly available sources, re-run lab experiments, etc. when possible.”
Coming from a completely different direction, theoretical statistician Larry Wasserman (link from Xian) suggests abandoning peer-reviewed journals entirely and replacing them by the Arxiv, a system run by physicists where people can upload their own articles. Arxiv is somewhat restricted (you have to be connected in some way to post an article, and they do enough screening so that, for example, they at first refused to publish my zombies paper (and, even when they did publish it, they removed George A. Romero from the list of authors)), but they don’t do anything like refereeing.
As Larry puts it:
The refereeing process is very noisy, time consuming and arbitrary. We should be dissem- inating our research as widely as possible. Instead, we let two or three referees stand in between our work and the rest of our field. . . . We should think about our field like a marketplace of ideas. Everyone should be free to put their ideas out there. There is no need for referees. Good ideas will get recognized, used and cited. Bad ideas will be ignored. . . .
As Xian points out, there are problems with Larry’s proposal in that it relies so strongly on the Arxiv and on personal websites. Larry’s suggestion of daily scanning of the Arxiv daily is not so practical—and it would be even much less so if his plan kicked in and the Arxiv suddenly started including the tens of thousands of papers outside of math and physics that are daily submitted to journals. Currently, journals serve as a filter for busy researchers and evaluators of research.
I think Larry would respond to this criticism by arguing that a no-journals system would create an incentive for groups of scholars to manage a filtering service. For example, instead of the American Statistical Association running JASA, JEBS, JBES, Technometrics, etc., and maintaining a separate editorial staff for each (representing a huge amount of volunteer service on the part of editors and referees), they could run a set of filtering services. The editors of each filter would be expected to scan the literature and handle submissions (which in this case would be pointers to articles already published on the web). The editorial boards would have the responsibility to come up with monthly (say) recommended reading material. Doing this would require some work, but less than the existing job of producing a journal. The main concern I see would be to keep the editors focused on solid research rather than getting tabloidlike. It would be tempting for an aggregator to give pointers to “pathological science” such as the Bible Code and silly sex-ratio analyses and ESP studies, in order to grab more attention. But as long as these aggregators took their jobs seriously, I’d think they could supplant journals.
Putting all this advice together
Considered separately, both Nyhan’s and Wasserman’s recommendations makes sense to me. But it’s striking that they go in opposite directions! Nyhan recommends a more rigorous system, where to publish an article you have to supply data and other replication materials, whereas Wasserman would place no restrictions at all.
I don’t have any answers here, it’s just interesting that two reasonable-sounding reforms of the current system can be so different. I can well imagine someone reading Nyhan’s suggestions and saying Yeah, then reading Wasserman’s suggestions and agreeing to those too, without even fully realizing their different directions.