Snotty reviewers

I had a submission a couple years ago that was rejected by a journal. One of the reviewers began with the following snotty aside:

In this manuscript Gelman and Shalizi (there’s no anonymity here; this thing has been floating around the web for some time) . . .

Actually, we posted it on the same day we submitted it to the journal. But double-blindness allowed the reviewer to act as if we had done something wrong! And, even if it had been “floating around the web for some time,” that’s not necessarily a bad thing. Perhaps it just meant that the article had previously been rejected by a bad-attitude reviewer!

28 thoughts on “Snotty reviewers

    • I had a reviewer reject my paper because the bibliography didn’t have enough entries to prove that I had done adequate literature review to check whether it was original. (Should I add a new section where I go through journal backlogs? “In [1], a completely unrelated result was found, with no resemblance to the current result. In [2], the same. In [3], yet again the same. In [4], a theorem was found which started with the same word as our main theorem, but aside from that the theorems bear little resemblance. In [5], …”

  1. I don’t understand. It seems to me, it’s much better that the reviewer did acknowledge that s/he has seen the paper (no matter how), instead of pretending double-blindness. And we, the readers of your blog, have no idea what made this reviewer particularly harsh on you, since you only posted the first sentence of the review (which in itself is neutral) and did not post your manuscript.

    I mean there are papers (recently Jager and Leek) that you negatively comment on. Thanks to your popularity, many reviewers would view those papers with a bad attitude. To clarify, a bad attitude may come from your fast and loose perspective, not from simply posting it on arXiv.org.

    • Anon:

      I have no strong feeling for blind review one way or another. My problem was not that the reviewer knew it was our paper. My problem was with the phrase, “this thing has been floating around the web for some time,” which sounds negative to me, as if they’re dismissing it for being old news. I’m not sure if it was on Arxiv, but it might have been.

      Of course I don’t object to people commenting on my posted papers! I love getting comments. What I don’t like is a reviewer disparaging a paper on the grounds that it was posted on the web, as if I’m supposed to keep it a secret until he or she finishes reviewing it.

      Regarding your other comment, I don’t actually know what you mean by “fast and loose,” but if you have any particular problems with my paper with Shalizi, please let me know.

      • Isn’t “double blind” better than not double blind?

        Knowing all about biases, prejudices etc. how can you be ambivalent about blind review?

      • Seems like a simple factual statement to me – the reviewer knows the author because the paper has been floating around the web for some time. You find it negative because you are, understandably, annoyed about the rejection, simple as that.

        In Marketing many top journals will not accept papers that have been “floating around the web”, btw.

        • “In Marketing many top journals will not accept papers that have been “floating around the web”, btw.”

          Which is definitely a problem and should be corrected ASAP.

        • We almost ran into this problem with a recent forthcoming paper. Because it was posted on SSRN as a working paper and very likely “more than 70%” was the same, we had to jump through hoops and allow additional consideration by Sage editors to ensure that we weren’t breaking any rules.

          And this was an Econ journal, a discipline where working papers are pretty much the standard. Most of the stuff Freakonomics picks up on are working papers. We were a bit blind-sided by this. Most of us know the double-blind thing isn’t a reasonable assumption in any case, so why keep up the front?

          My wife is a neuroscientist and none of the papers she submits are double blind. Reviewers always get to see the author names.

    • Critical reviews should always be helpful to the authors if they process the information thoughtfully – but they should do that as a matter of habit. Pretend not matter how hard it mat seem, that the reviewer was trying to help you. If they have misunderstood something, how can you put it so that they would be less likely to misunderstand.

      In some of the (blinded) reviews, I have been very negative (e.g. the work was embarrassing to review, anything worth doing is worth doing well, disapointed that an author with this background would suggest, etc.) but hopefully respectful and polite.

      Usually, the revised papers are much better (you need to indicate what you feel needs to be addressed carefully) and in a couple cases surpisingly better.)

      IMHO Andrew did the authors a great favour – they just do not realise it yet.

  2. Is double-blind review standard practice in the social sciences or was it just the practice of that particular journal? In the physical sciences reviewers are anonymous but authors are not. Double-blind seems a far better arrangement to me.

    What’s the downside to circulating a manuscript before submitting it to a journal? (From the standpoint of promoting good science, that is.) It’s an opportunity to get feedback, make revisions, and create a better end product.

    • Chris:

      I’ve seen it done both ways. In a field such as econ, it’s standard to workshop a paper all over, and in that case double-blind is more of a formality than anything else.

      • In my discipline (sociology) double-blind is increasingly a formality. In fact if you are about to submit a paper for blind peer review, you’d have to work pretty hard to make it difficult for a curious reviewer to figure out authorship — different title from any public presentations, being careful not to have a the text or even abstract of any earlier version on the web, etc.

        On a related topic, a sociology journal recently began adhering to the following policy, which they communicate to reviewers:

        “Please be advised that the [journal name removed] editorial team reserves the right to edit reviewer comments for authors to assure consistency of message, to correct obvious errors, to remove incendiary or ad hominem comments, and to otherwise ensure a seamless flow of information to our authors.”

        Most of my colleagues were outraged by this and have announced that they will not review for this journal. But I think the policy is reasonable. In a situation like yours, an editor might decide that a comment like “floating around the web for some time” is irrelevant and possibly unprofessional. It serves no purpose to have the author read and anticipate reacting to that comment in doing a revision. Or the editor might decide that the reviewer’s comments and recommendations for revision are totally off the mark. In almost all sociology journals, reviewers see the editor’s letter to the author (along with copies of the other reviews). As a result, editors would be reluctant to write something like “you can ignore reviewer C because it in my judgment his comments are not only not helpful, they are just plain wrong.” Instead, they write something diplomatic like “in your revision, pay particular attention to points 1 and 2 of reviewer A and 2 and 3 of reviewer B. But a young scholar might find it hard to ignore highly critical, nasty, but off the mark comments like those from a reviewer C. Telling reviewers up front that their comments might not be sent verbatim to the author could be a diplomatic way out of this dilemma — particularly when the editor recognizes that reviewer C was way out of line on this manuscript but is usually quite helpful and will be asked to review again for the journal.

      • If a journal’s policy is indeed double blind then releasing / circulating a paper prior to review seems circumventing the system.

        Shouldn’t journals that use double blind review have a rule about not accepting papers for review that’ve been publicly released?

        • This circumvents the goal of an open and “real time” exchange of ideas, which seems like it should be the top priority of research. Maybe not in marketing, which is, after all, a discipline devoted to getting people to buy stuff even if they don’t need it, want it, or could get it for free elsewhere.

          If time in review was on the order of 2-3 weeks, I’d be less concerned about a policy that papers under review couldn’t be posted on the web. But it isn’t: in some fields, total time in review is closer to 2-3 years. Tell me how quashing access to ideas for 2-3 years is good for science? Good for Elsevier’s business model, maybe…

        • Rahul:

          If any journal wants to have a policy of not accepting papers for review that’ve been publicly released, they should announce that policy officially. That will make it easier for every economist in the world, every physicist in the world, and lots of statisticians (including me) to know ahead of time never to submit there.

        • I don’t know about stat n econ, but in physics they don’t pretend to go double-blind even.

          My point is, you can have double-blind or have free dissemination but not both. If you want public release the double-blind part is a charade.

  3. Blind review is often not blind. In more than a third of the papers I’ve reviewed, I’ve figured out who wrote it without difficulty (some have a distinctive style, some you know are working on that exact problem with that line of approach, some people can’t help but fill their references with mentions of their own work – I could mention one or two who are particularly fond of it). But even when I’ve had a copy of the working paper before I get the review copy (and thus knew who it was by before I finished reading the title), I still wouldn’t write something like that. I’ve written a few somewhat snarky reviews (well deserved), but if its meant to be double blind I at least try to behave like it is.

      • Not least, to try avoid any sense of what generated the post. *I* know I’m trying hard to be totally impartial even when I realize whose paper it is, but the authors don’t. If you make it clear you know who wrote it, an unintentionally infelicitous wording nearby may give a particularly dire impression.

  4. As always there are two sides to look at. Authors have an interest in gaining the maximum exposure for their work through quick dissemination. Journals have an interest in restricting access to those that have paid for it.
    I’ve had the experience of submitting a paper to a journal and being told that it would not be sent out to referees until the working paper version of it was taken down from my web page. Fair enough, if I want to submit to that journal I have to play by their rules (even if I think the rules are senseless). However, I think this sort of rule should come with a responsibility to expedite the acceptance/rejection process within a reasonable timescale. After almost 18 months I’m still waiting for a decision, which means, even if the piece is accepted, the timescale from submission to acceptance will be about 3 years! Keeping work out of the public domain for such a long length of time seems to me to be an odd way to do science.

  5. Regarding “double blind is always better”, what counts against it is, as was mentioned before, that “double blind” may not really be double blind because some reviewers can guess the authors (and some papers are around before being published), so it suggests a degree of secrecy to the novice that isn’t really true. “The reviewer will know the authors in any case” is more honest.

  6. Surely ‘double blind’ can be helpful in a subset of cases where the authors are little-known in the field, and will not be recognized, but don’t want the potential stigma of being little-known.

  7. if you are going to get upset each time a referee does something stupid, obnoxious or insulting, you won’t have enough time left to post to this blog, let alone time for research, eating, etc

    At least it isn’t out and out dishonest, like sitting on your paper while one of hte referee’s students gets out a competing paper.

  8. Thank you! @stringph … double blind ensures that authors without status can avoid their low-status penalty by not making their paper public and keeping the review “blind” … I am amazed by teh pretentiousness of some senior scholars who want science to be judged by its merits yet assume that their fame/status will have no impact on acceptance decisions. Prof. Gelman might have posted the paper online (arxiv or wherever) for dissemination purposes – but there are several scholars who would like to ride their status for getting articles accepted at journals: as long as one can’t tell who has what motives, double blind review still serves the purpose, however minor in your estimation, of allowing fledgling/unknown scholars to have their paper judged for its content alone rather than their name/race/gender/university affiliation.

    • >Thank you! @stringph … double blind ensures that authors without status can avoid their low-status penalty by not making their paper public and keeping the review “blind” … I am amazed by teh pretentiousness of some senior scholars who want science to be judged by its merits yet assume that their fame/status will have no impact on acceptance decisions.

      Second that and the comments that follow.

  9. When I was a journal editor, I quickly realized that (a) not all reviewers are politic and helpful and (b) you have to husband your most important resource, reviewers. If you don’t pay attention to the care and feeding of good reviewers, you end up with lousy feedback for authors. My own policy was to suggest to reviewers that they temper their remarks and consider how they would feel getting slammed. It (mostly) worked.

  10. Pingback: Philosophy and the practice of Bayesian statistics (with all the discussions!) « Statistical Modeling, Causal Inference, and Social Science

Comments are closed.