Chris Said points us to two proposals to fix the system for reviewing scientific papers. Both the proposals are focused on biological research.
The growing problems with scientific research are by now well known: Many results in the top journals are cherry picked, methodological weaknesses and other important caveats are often swept under the rug, and a large fraction of findings cannot be replicated. In some rare cases, there is even outright fraud. This waste of resources is unfair to the general public that pays for most of the research. . . .
Scientists have known about these problems for decades, and there have been several well-intentioned efforts to fix them. The Journal of Articles in Support of the Null Hypothesis (JASNH) is specifically dedicated to null results. . . . Simmons and colleagues (2011) have proposed lists of regulations for other journals to enforce, including minimum sample sizes and requirements for the disclosure of all variables and analyses.
As well-intentioned as these important, necessary first steps may be, they have all failed to catch on.
Before getting to Said’s suggestion, let me interject that, yes, I agree these are important problems, but I don’t like the idea of a “Journal of Articles in Support of the Null Hypothesis.” I think the whole null hypothesis thing is a bad idea. I understand the appeal of seeing whether a pattern can be explained merely by chance, but let’s not go overboard here: in lots of examples of biological and social sciences, the null hypothesis can’t be true. The issue isn’t “support of the null hypothesis” so much as inferential uncertainty.
Granting agencies should reward scientists who publish in journals that have acceptance criteria that are aligned with good science. In particular, the agencies should favor journals that devote special sections to replications, including failures to replicate. . . . I would like to see some preference given to fully “outcome-unbiased” journals that make decisions based on the quality of the experimental design and the importance of the scientific question, not the outcome of the experiment. This type of policy naturally eliminates the temptation to manipulate data towards desired outcomes.
The recommendations seem reasonable but I disagree with the claim that, if implemented, they would “eliminates the temptation to manipulate data towards desired outcomes.” We’re doing science here, we want to make discoveries! Of course there is a temptation to find what we want to find. I agree that if publication in the “tabloids” and grant funding aren’t on the line, the temptations are lower, but they’re still there.
Said also points to this proposal by Niko Kriegeskorte for open post-publication peer review. I don’t quite see how it works but it’s probably a good idea, sort of like my modification of Larry Wasserman’s idea mentioned in our previous post, where I note that if all our publication shifts to Arxiv-like repositories, the defunct journals can retool as lists of recommended reading.
P.S. Said also writes that, “in the current system, the only signal of a paper’s quality is the journal’s impact factor.” I don’t see this at all. Here are some other signals:
- Journal quality. Impact factor != quality. For example, mediocre biology journals have higher impact factors than top statistics journals.
- Citation counts. With Google you can start counting citations right away. Citations are no guarantee of quality either, but they are another signal, not the same as the journal’s impact factor.
- The authors. All else equal, I’d expect a paper from a recognized lab to be taken more seriously than a paper from nowhere.
- And, of course, the paper itself, starting with the title and abstract.
That’s a lot of signals right there (see also also the many different ratings collated here), and I’m probably forgetting a few more.
As I wrote in the previous post, I’m glad to see people thinking about reforms. I made the comments above not to shoot down the ideas of Said and Kriegeskorte but rather to explore some complications.
The current system has obvious problems; one result of this is almost anything can seem like a good solution. It’s sort of like education reform: choose Back to Basics, or Student-Centered Learning, or whatever: any of these ideas could be good, but it depends on their implementations. In any case, Said is probably right that funders could push for a lot of changes. Sort of like what William F. Buckley said about college education in the 1950s.