Another perspective on peer review

Andrew Preston writes:

I’m the co-founder and CEO of Publons.com. Our mission is to modernise peer review. We’ve developed a way to give researchers formal recognition for their peer review (i.e., evidence that can go on your CV) and have partnerships with over 1k top journals.

I’ve just finished reading your rebuttal of Susan Fiske’s editorial critique of social media. While this is directly relevant to the discussion that happens on Publons (great example here), your description of what has happened in the last 10 years brings to mind a Kuhnian paradigm shift in the field.

I think that’s important.

We’re working on a course to train peer reviewers, which we’re calling the Publons Academy. An awareness of statistical issues is an important part of that.

I have no idea what this is all about, and as you know I’d like to get rid of the current reviewing system entirely, but I thought I’d share this with you.

21 thoughts on “Another perspective on peer review

    • Not money grab. I have been using Publons for a couple of years. What I do is forward them the “thank you” emails I get from the editors of the journals I have reviewed an article for. This results (at a minimum) in a record showing for which journals I have produced a review and for how many times. You can even report (and make it viewable to the rest of the world) the titles of the articles reviewed and even the reviews themselves, but this is entirely optional and not the default option. Good service, and free for the user.

      • But most people just list in their CV the journals they’ve refereed for. What is gained by using Publons? I suppose it is handy to have all this in a database, but… why? For example, as an editor, when I look for reviewers, I don’t really care who they’ve reviewed for in the past. I look at their publication record.

        Moreover, if the profession does not reward refereeing, that won’t change just because it’s listed in a database.

        • At the moment it’s just personal satisfaction for me. Like: “ok, I have spent 10 hrs reading and reviewing a paper. Allright, there is no glory in this and no one will ever reward me for the time I spent doing it. But at least somewhere it says that I did a review for journal XXX, and this can be certified”, as opposed to me stating it in my personal website. Again, small consolation. But who knows if one day things will evolve in a different way. It is really as simple as just forwarding the “thank you” email and filling up some metadata. Time invested: 60 seconds

  1. Thus far, I’ve ignored all requests to register my review with publons. If doing journal reviews are a currency then I have yet to figure out how or where to spend such currency. Until I am convinced otherwise I will view review credit like I view Monopoly money. Right now, I know that my colleagues, department chair, Dean, granting agencies, and journals don’t care one bit how much reviewing I’ve done.

      • I don’t think there is any specific “credit” given for reviewing. There seems to be a nominal expectation of some kind of “service,” that could include reviewing as well as other things, but I don’t think anyone ever got denied promotion, etc., for “poor service” (unless perhaps it was a matter of some other reason people wanted to get rid of the person, and “inadequate service” was an excuse).

    • I thought was reviewing was thankless until I discovered that the Hungarian Pastry shop near Columbia has this punch card, where for every five high impact factor reviews you do you get a free piece of baklava. You should see if your local pizza place does something similar. You’d be surprised!

    • “I know that my colleagues, department chair, Dean, granting agencies, and journals don’t care one bit how much reviewing I’ve done.”

      If this is in fact true in the general case, then I believe we can show that the total added value of traditional peer review cannot be any higher than direct value paid to reviewers (which I take to be asymptotically approaching zero), plus intangibles like the self satisfaction of the reviewers, the learning value of being a reviewer (which is hard to see being very large given one has access to all the published papers anyway), and the value of the quid pro quo.

      Conditional on Marcus’ assessment being generally true, I agree with Andrew that this doesn’t make a very compelling case for the value of the traditional system.

      • Kevin:

        +1. As I wrote in my earlier post, one of the big problems with the current peer review system is its gross inefficiency: the vast majority of review reports are for papers that just about nobody will read or care about. Meanwhile, there are a few papers that get attention, but most of the reviewing of these will be done after the papers are published, and there is no place for post-publication review in the traditional system.

  2. I’m the co-founder and CEO of Buggons.com. Our mission is to modernise the buggy whip. We’ve developed a way to give coachmen formal recognition for their buggy whipping (i.e., evidence that can go on your CV) and have partnerships with over 1k top coach lines.

    I’ve just finished reading your rebuttal of Michael Link’s editorial critique of motorized coaches. While this is directly relevant to the discussion that happens on Buggons, your description of what has happened in the last 10 years brings to mind a technological revolution in the field.

    I think that’s important.

    We’re working on a course to train buggy whippers, which we’re calling the Buggons Academy. An awareness of mechanical issues is an important part of that.

    • Lack of training in reviewing papers has been a large problem that likely continues but I think incentives to review are likely (or should be) disappearing exponentially.

      Journal publication makes less and less sense in any communal process of doing science purposefully.

      Something needs to replace it – an overhaul not repairs here and there.

  3. “We’ve developed a way to give researchers formal recognition for their peer review (i.e., evidence that can go on your CV) and have partnerships with over 1k top journals.”

    If you are a professor, i believe that in many cases you already get formal recognition for your peer review.

    This comes in the form of “contributions to revising the manuscript” of their (PhD) students and/or friendly colleagues.

    It’s usually listed under “publications” on their CV’s.

Leave a Reply to Carol Cancel reply

Your email address will not be published. Required fields are marked *