Hey, here’s a new reason for a journal to reject a paper: it’s “annoying” that it’s already on a preprint server

Alex Gamma writes:

I’m interested in publishing in journal X. So I inquire about X’s preprint policy.

X’s editor informs me that

[Journal X] does not prohibit placing submitted manuscripts on preprint servers. Some reviewers may notice the server version of the article, however, and they may find the lack of anonymity so annoying that it affects their recommendations about the paper.

This is interesting in part because it highlights the different roles of scientific journals. Traditionally, a journal is a way to “publish” a paper, that is, to print the article so that other people can read it. In this case, it’s already on the preprint server, so the main purpose of the journal is to give a stamp of approval. This all seems like a mess.

I think it’s inappropriate for a reviewer to downgrade a submission because it’s been posted on a preprint server. I don’t know about the rest of you, but sometimes I do something that seems important enough that I don’t feel like waiting 2 years for it to appear in print!

At the very least, I feel that the editor, in his or her instructions to reviewers, could explicitly instruct them not be influenced by the existing publication history. This stricture can’t be enforced but at least it could be established as a norm. As it is, it almost sounds like the editor thinks this attitude on the part of reviewers is OK.

44 thoughts on “Hey, here’s a new reason for a journal to reject a paper: it’s “annoying” that it’s already on a preprint server

  1. “…and they may find the lack of anonymity so annoying…”

    Some journals have a double-blind reviewing policy: when the paper is sent to reviewers the authors’ names are removed, and authors are asked to also remove anything in the paper that might easily identify them. The aim is to prevent reviewers’ opinions about the authors from influencing their review of the paper. I wonder if the issue here is that a preprint subverts this policy. Personally, I don’t find the double-blind policy to be helpful either as an author or a reviewer, but others might.

      • There’s an excellent recent paper on blinded peer review: “Reviewer bias in single- versus double-blind peer review,” Tomkins et al. PNAS 114:12708–12713.

        “We present a controlled experiment in which four committee members review each paper. Two of these four reviewers are drawn from a pool of committee members with access to author information; the other two are drawn from a disjoint pool without such access. … Once papers are allocated to reviewers, single-blind reviewers are significantly more likely than their double-blind counterparts to recommend for acceptance papers from famous authors, top universities, and top companies. The estimated odds multipliers are tangible, at 1.63, 1.58, and 2.10, respectively.” Interestingly, they found a small bias in favor of female authors if author names are revealed (i.e.not blinded).

    • “I wonder if the issue here is that a preprint subverts this [double-blind] policy.” Indeed, this has come up before.

      From https://www.timeshighereducation.com/news/reviewer-criticises-no-publication-after-preprint-rule :

      “Professor Ha, professor in the department of media production and studies at Bowling Green State University, said that prior online publication damaged the integrity of the peer review process because reviewers could identify the author through online search. Authors who submit to the journal are required to confirm that the manuscript has not been published elsewhere, she said.”

      “‘Posting research articles online while submitting to refereed journals is a threat to the double-blind review process and journal publishing,’ Professor Ha said. ‘We respect authors’ choices to post their own works online for free public access without going through the referee process, publish in free non-profit open-access journals or pay for the publishing fees of open-access journals.”

    • I thought this was the norm in most fields. Whatever the costs and benefits, it’s best to play by the rules so it works right. With that said, the reviewer should in most situations have quite a bit of control over whether they are blinded. For instance, I would consider it wrong to do a Google search of the paper’s title to look for pre-prints of it. Obviously some papers get shared on Twitter and such and you could accidentally find out the identity of the authors, in which case you just have to do your best to be unbiased and judge it on the merits.

      I’m a PhD student so I know that someone finding out I’m the author of a paper they’re reviewing would likely work against me. While the CV I give to my program’s annual reviews and such includes works under review, I don’t publish that online so I don’t have to worry about it. And of course in my field it’s unusual to publish pre-prints. An advisor tells me part of the reason, other than our field just not being on the cutting edge of this stuff, is that 10 or so years ago some of the top journals started rejecting papers that were previously submitted to our top conference since the conference freely shared the drafts through a web portal. The journals went back on their policies after the resulting backlash — and the conference made the drafts available on an optional basis and for only a limited time — but people are still gun-shy.

      • I google the papers I review to make sure they are not plagiarized (or more often, salami sliced from other papers the authors have already published). Those are critical things for a reviewer to note!

        As a reviewer I would guess maybe a 1/3 of the papers I review I could guess the authors with a high probability (even without googling). Presuming it would work the same way for my work, even if I did not post pre-prints and present work at conferences, my anonymity as a writer would be a farce in a non-trivial number of cases anyway. Since I prefer more than a dozen people hear about my work though I do those things.

        As a writer, I really don’t care about losing my anonymity though. I know some reviewers are hostile, but I do not think it is because they don’t like me personally. Even if they don’t like me and level legitimate complaints I need to respond to them. If they are not legitimate or are trivial a good editor will take that into account. (And the editor knows your identity – which is the person who makes the most important decision.)

      • By “this was the norm” do you mean double-blinding?
        I don’t know what fields do that. I’d think a small minority.
        Can readers name the fields that do for me?

    • One issue with double blind reviewing is that some reviewers may guess the author easily anyway, or try to guess, so different reviewers will make different assumptions (or not) in a wildly random manner.

  2. This is funny because in other fields, like mine, it’s the opposite. A reviewer may be skeptical or annoyed if a paper has not already “made the rounds” via conference circulation, preprint server posting, etc.

  3. To all:

    Just to clarify: Yes, I can see that there could be good reasons to require double-blind review, which indeed is incompatible with previous publication via preprints. And I can see how a journal might then state as a policy that they won’t review any paper that has been previously published in any form, preprints included. What seems odd to me is for them to not have such a policy but to then allow their publication decisions to depend on reviewers finding a paper “annoying.” I don’t think that being annoyed is a good reason to reject a paper. Nor, for that matter, do I think that not being annoyed is a good reason to accept a paper.

    • But then shouldn’t that also exclude any papers that appear in conference programs? Anyone googling the title can identify the authors. This comment is meant as a reductio.

  4. I could get behind David Colquhoun’s “Just let it go out of business quietly.” I’m also with Thomas Basbøll’s response of “Obsolete?” to “so the main purpose of the journal is …”

    Andrew must have lived through very different traditions than me if he can say, “Traditionally, a journal is a way to “publish” a paper, that is, to print the article so that other people can read it.” The primary purpose has always seemed to me to be peer review in order to rank professors for tenure and promotion based on the journals into which they were able to place their papers. I don’t see that it’d exist otherwise. Same purpose as grades in school or college. If they didn’t grant degrees that people cared about, I think it’d be hard to motivate most people to come for the learning alone if it could be achieved by other means (like mentoring). Citation indexes are just taking this tendency to the logical conclusion.

    The first step to reform is something like Transactions of the Association for Computational Linguistics. It’s an open-access, free-to-publish, peer-reviewed journal with fast turnaround times (on the order of month to first decision without a tendency to request endless revise-and-resubmit rounds). This cuts down tremendously on the embargo period of reviewing, which is just absurd. Hold back results for two years? Not why I signed up for academia. I’d go back into industry if I wanted everything I did to be bound up in non-disclosure. I would list something like BMC Bioinformatics for having the same goal and fast turnaround time, but that’s a pay-to-publish operation, which we honestly don’t need. Publishing an online journal is very cheap. I’d list JSS for the open-access, free-to-publish part, but they’re very slow (and oddly demanding on formatting for a journal that uses a Dali-esque painting as a logo!).

    Double-blind reviewing doesn’t work very well in my experience. Engineering and lab set ups and style are so distinctive that it’s usually pretty easy to track the provenance of the majority of papers, even with redacted bibliographies. You may not be able to list the reviewers, but you’ll know it came out of so-and-so’s lab at Stanford, because they’re the only people combining data sets like A and techniques like B. And even if you’re wrong, the guess weighs almost as heavily on reviewing decisions. On the other hand, if big labs are getting breaks, we can’t make it optional, or the big players would just be open to exploit their advantage.

    Even in the early 1980s, communication didn’t seem the main purpose. We had conference papers and preprint servers that did that much better even before the world-wide web. It might’ve been more useful for dissemination before the internet and easy global travel.

    I’ve never understood the “part of the permanent record” business. Academics seem to have a *very* short memory in y experience and basically zero desire to trace back through original sources for reasons other than writing history papers. Nowadays, I think keeping distributed backups of arXiv that you could buy on disk drives would be plenty of permanence. I don’t think we need the dead tree part any more.

    Where does talking about a paper at a conference fit? How about a talk? What if the reviewer saw the speaker talk about it? How many reviewers would then recuse themselves? The whole business is a huge conflict of interest starting from not wanting competitors to get exposure through publication and shutting out competing ideas (not the same thing—competitors usually work on similar ideas rather than competing ideas).

    • Bob:

      Perhaps our difference is that you started research in academia and I started research in non-academic labs (first the Naval Research Lab and then Bell Labs). In those places, the purpose of presenting at conferences, and of publishing papers, was to share results with the relevant research community. Sure, there was also some goal of getting credit for our contributions and getting glory, which in turn could translate into a promotion or higher salary or simply not getting fired—but I think the main goal was to just get the ideas out there, just as we do now by blogging or writing blog comments. And back in the 1980s there were no preprint servers. Some groups did have tech report series that they would send out, but these were much less accessible than journal articles.

      • I was using preprint servers in 1984 as soon as I started grad school in Edinburgh. We had FTP and we had Usenet. Maybe preprint servers didn’t catch on in stats until much later.

        In computer science, conferences are the main publication venue, so they’re used for attribution/status as well as getting ideas out there.

        Tech reports weren’t any less accessible than journals at top institutions. In addition to online versions, we’d share print versions with like-minded institutions like Stanford. We’d pass them out and read them, or photocopy them if they were good and redistribute. Everything of interest was being done out of the main journal publication loop, which was ridiculously slow even by FTP and Usenet and conference dead tree proceedings standards.

        • “We live in an age of science and abundance. The care and reverence for books as such, proper to an age when no book was duplicated until someone took the pains to copy it out by hand, is obviously no longer suited to ‘the needs of society’, or to the conservation of learning. The weeder is supremely needed if the Garden of the Muses is to persist as a garden.” (Ezra Pound, ABC of Reading, 1934, p. 1)

    • I’ve never understood the “part of the permanent record” business. Academics seem to have a *very* short memory in y experience and basically zero desire to trace back through original sources for reasons other than writing history papers. Nowadays, I think keeping distributed backups of arXiv that you could buy on disk drives would be plenty of permanence. I don’t think we need the dead tree part any more.

      Despite the technological limitations of the time, I’ve found pre-1940s literature to be extremely quality dense (relative to today…). I would not recommend throwing it away or making it at all more difficult to access. In fact I think in the case of biomed (and probably psychology, etc), a project that goes back and implements ideas from that era with modern computing could easily be more productive than the status quo, and for much cheaper too.

      • I always suggest going back to the 80s literature to find all the marginalizations we need for Stan. They were being used for EM at the time.

        I find the older literature amusing in that it’ll include tricks for arranging file folders in the back of a station wagon (Cochran’s survey sampling book) or thanking students for their work on slide rules (Mosteller and Wallace’s Federalist papers books). But the newer literature is much easier to digest. And I think it’s getting better, much like other writing such as TV and novels getting better (at least in the fantasy and sci-fi genres I watch and read).

        Having said that, I’m the one who wrote into a survey of best papers in natural language with (Shannon 1948), which pretty much defined the modern approach and included things like posterior predictive checks. I think everyone on the thread thought n-grams were invented in the old days (1990s), not ancient times (1940s).

    • I think the solution lies in the hiring and tenure committees. They must reassert their competence to assess the scientific competence of candidates by looking directly at their work and the conversations they participate in.

      I don’t see why a system based on open institutional repositories couldn’t work. Everyone just uploads their stuff and notifies their networks. I actually think the Science Citation Index could continue to provide a useful (and profitable) service even after the journals have all disappeared, since the links between papers in institutional repositories would be of interest also to scholars reviewing the literature.

      There would also be room for a very few quarterlies of stellar work (culled from the repositories) and reviews containing commissioned articles by major scholars.

      • For the departments where I’ve seen Ph.D. admissions done at Carnegie Mellon and Columbia, it’s been very selective. Good departments are drowning in candidates with great grades and test scores. Often admissions criteria include balance—you may have 20 great students applying to do systems, but you want a balanced class of theory, systems, hardware, software, etc. It’s not as simple as ranking the students and take the top-N.

        The goal in admissions has never been “cheap labor” in my experience at CMU or Columbia. Grad student labor is expensive when you have to shell out nearly $50K in tuition and you don’t even get skilled, full-time labor in return! Postdocs are much better value and research scientists like me even better value still because we can apply for funding ourelves. At CMU, the goal was sometimes to admit enough people to fill known RA positions. But I don’t recall anyone ever being admitted specifically because they matched a product that needed labor. For Columbia stats, the students are pretty much all TAs, which makes them very expensive teaching labor.

        Everyone wants grad students that’ll be productive by regular research metrics, which means papers. That’s win-win for the student and faculty member. Especially if the student gets a good job. That’s how broader influence in ideas gets spread and that’s a much bigger deal than cheap labor. Prestigious journals aren’t the only way to spread the word—it’s just a very effective way that works for people who don’t know the field (as Jordan points out in his blog post).

    • Terry:

      It’s not at all clear, given that the journal editor gives no guidelines other than to indicate that he or she is open to having decisions being made based on referees being annoyed.

  5. Since we’re talking about a journal’s attitudes towards preprints I can’t help but take this opportunity to point out the hypocrisy of Cell Press.

    Cell Press was always unofficially hostile towards preprints, but they made their position clear in a blog post where they fearmongered about preprints: http://crosstalk.cell.com/blog/lets-talk-about-preprint-servers

    But then they ended up making their own sort of preprint server: https://www.mendeley.com/sneak-peek/cellpress/

    Then Emilie Marcus showed up at the ASAPbio meeting and said that not only did Cell Press support early dissemination of research in the form of preprints, but that they wanted to explore ways to get research out in an even earlier form. https://www.youtube.com/watch?v=9YWnulF3ank

    So they went from not liking research shared via preprints, to publicly claiming they wanted research shared even before preprints.

    And when Emilie Marcus left they listed one of her achievements as the preprint server: https://www.elsevier.com/about/press-releases/corporate/cell-press-ceo-and-editor-in-chief-of-cell-emilie-marcus-to-pursue-new-career-at-ucla

    I don’t understand when people try to change the narrative like this. Do they think we have terrible memories?

    • Jordan:

      I’ve seen so much behavior that seems not only immoral but also seemingly counterproductive in what I imagine would be the effect on the reputation of the person doing the bad behavior. Yet this keeps going on, and in many cases the evildoers seem to be rewarded in net. Their reputation falls among the subset of people who are in the know, but outside that circle, the bad behavior is not recognized.

      I can only conclude that:

      1. Yes, people have terrible memories.

      2. There’s lots of churn: every year, many people leave the system and many others enter. So, even if individual memories are long, collective memory is short.

      3. Some subset of people out there are themselves cheaters, and they may well have a “But for the grace of god, there go I” mentality and thus are loath to condemn bad behavior in others.

      4. Reputational news is mostly second hand and is not always trusted.

      5. It can be possible, indeed easy, for people to muddy the waters. Maybe the critic has a political agenda or is just a hater? Maybe the critic is simply jealous of others’ successes—or, conversely, is nothing but a bully? Etc.

      6. There can be legitimate disagreements on the merits. I don’t know enough about your example above, but typically there is some reasoning by which apparent bad behavior is actually ok.

      We can probably come up with a few more reasons to add to this list . . .

        • Jordan:

          I think a key challenge here is harnessing all the free effort that people put in when reviewing for Cell and all those other journals. We’ve been told that it’s good citizenship to review papers for journals for free. Proposals to replace closed journals with open peer review have had difficulty because very few people want to do open peer review for free. Maybe that’s fine—we’ll just reach a new equilibrium in which people publish more of their own stuff and do fewer reviews of others’ work. If reviews of others’ work is truly valued, organizations will pay for it, either directly or through influence on jobs, promotions, etc. This is not so hard to imagine, as referee reports are a form of “review article,” and review articles are considered to count for something.

    • From the first cell link:

      Science advocates all over the world are working tirelessly to help non-scientists, especially policymakers, to understand the importance of peer review, evidence, intellectual rigor, and credibility in the scientific community.

      So many problems here…

      1) Conflating peer review with evidence, credibility, etc.

      2) By peer review they mean only institutionalized peer review, specifically not people informally getting feedback from each other.

      3) Assumes a paper can either be peer reviewed or not peer reviewed, ignoring that some drunk post-doc glancing at your paper for an hour can count as institutionalized peer review.

      4) There is plenty of evidence that institutionalized peer review has negative side effects, or even that its primary role is actually to enforce whatever prevailing biases exist: https://en.wikipedia.org/wiki/Scholarly_peer_review#Allegations_of_bias_and_suppression

      5) The modern role of institutionalized peer review is a relatively new phenomenon, lots (I’d say even much better) science got published before it became the norm: http://michaelnielsen.org/blog/three-myths-about-scientific-peer-review/

      This is all pretty obvious and/or well known to anyone who has done a bit of research on the topic, so I have trouble taking that post as genuine. I didn’t bother with the rest but it wouldn’t surprise me if the position was changed based on market sentiment or whatever since I don’t believe there was any real principle/belief behind it.

  6. I think arXiv (and similar) is the future. This defeats double-blind review, so I think we need to come up with a different solution to the problem of bias against women and people from less famous universities. One idea might be to have a “board” of established researchers whose job is to select a random sample of non-famous-author papers, identify the good papers, and publish prominent reviews of them. Such a board could not review everything, but they could monitor the preprint servers. As an arXiv moderator myself, I could easily point out papers that look potentially interesting.

    My experience with peer review the past few years is that it mostly functions as a mechanism for (a) identifying which parts of your paper people do not understand and (b) forcing you to ask yourself how you might be able to improve the paper. The literal content of the reviews is rarely useful.

    • As a an arXiv moderator you are already aware that arXiv itself is biased in favor of professors and PhD’s from so called top universities, as the genuine papers from outside the so called elite community rarely get approved by moderators to be put up on arXiv which is not even peer reviewed.

      ArXiv just cam not be the future, because that would be akin to cult proliferation. The moderators of arXiv from mostly reputed American universities, effectively allowing their own community to put up any papers, however technically flawed, on the arXiv, and then blocking papers from outside their community without giving valid reason, and citing “arXiv has right to reject any paper without giving reason”.

      No unfair , non-transparent, and arrogant system (like arXiv) can ever be the future!
      Peer reviewed journals are way better , despite their own flaws, because they is at least some system in place to ensure personal biases don’t dictate moderation decisions.

  7. On the Open Science Framework you can ask for it to be double-blinded – there is a link for contributors (i.e. reviewers) which does not reveal author names for peer review). However, I’ve used it to create a pre-print on a repository PsyArVix and that doesn’t allow the option of removing all authors for citation (yet). Currently enquiring about whether this is possible (and if it is possible to have a link stating this article has been submitted for publication in an academic journal so citation data is not currently available) as it could satisfy those reviewers who look for copies of research on the open web.

    Seems that Elsevier are a big reason why pre-prints have not taken off yet, by reinforcing associations between publishing in academic journals and scientific merit, but even they’re changing their business model by going for big data on scientists instead.

Leave a Reply to Raghu Parthasarathy Cancel reply

Your email address will not be published. Required fields are marked *