Hey! Can you guess the 3 goofy tricks that this new journal is trying to improve peer review?

Rob Bloomfield writes:

I [Bloomfield] am a new editor of a new Journal of Financial Reporting, and we’re trying to make some changes to peer review in our field.  I’d be very interested to hear your and your readers’ thoughts on whether our approach will help address the types of problems often discussed on your blog.

I’ve laid out the approach in two short videos and a more detailed pitch to authors (https://ssrn.com/abstract=2913262).

Below are the video links with short snippets of transcript. The full transcript is included on youtube for those who don’t care for video.

Substance, Commentary and Compromise:  JFR’s  New Approach to Academic Publishing (https://youtu.be/Qf8FkuBMfus)

Starting a brand new journal gives us a chance to make some real changes in how papers are written and published.

Tell me if you’ve heard these common concerns:

  • Journals ask for novelty, but they don’t treat it kindly.  They distort new ideas to fit old thinking, or simply reject them.
  • Journals ask for rigor, but don’t re-examine prior claims. They leave the first word the last word.
  • Journals promise a level playing field, but authors without connections and prestige find it hard to publish good work.

These problems aren’t secrets, but journals have little incentive to change once they are well established.  That’s where JFR comes in, and here’s our fix.

First, we recognize that every paper must compromise on some goals to achieve others.  It’s hard to explore new ideas and test old ones. It’s hard to identify causality and generalize across samples and settings.  It’s hard to address deep issues and provide useful applications.  We’re fine with compromises; in fact, we encourage them.  We’ll let you pursue a limited set of goals, as long as you do it well and report honestly about what you did and didn’t achieve.

Second, we’ll base publication decisions on the substance of your work—what you actually do and find.  How do you gather data? How do you analyze it?  What narrow conclusions do you draw? We’ll hold substance to the highest standards.  But most papers also include a good deal of commentary.  What makes the study interesting?  How does it address big questions?  How does it test or extend existing theory?  How might it affect practice or policy?  These issues are important, but they’re so hard to evaluate that a lot of good papers get rejected on the basis of reasonable disagreements about highly subjective matters.  We think that’s wrong, and that this is one of the big reasons peer review is so tough on new work and unknown scholars.  We won’t use reasonable disagreements as a basis for rejection.  Instead, if need be, we’ll hash them out in published discussions.

By allowing compromise and reasonable disagreement, we at JFR can make a much stronger push for innovation.

The Journal of Financial Reporting Playbook (https://youtu.be/E7mWo0P9qZ8)

We have a different view of the publishing process.  For any journal, you can think of a manuscript as a football; we score points by getting the ball over the goal line.   All too often, journals make authors feel like their battle is against reviewers and editors. But nature is the real opponent:  complexity, randomness and flawed data blur our vision and make it hard to measure and identify causes and effects.

Authors are the so-called ‘skill’ players who run, pass and kick the ball forward, by bringing new theory, data and analysis to the field. Reviewers aren’t working against the authors, they are helping them find a path through nature’s defenses.

Where are the editors?  We’re the coaches over on the sideline, making sure that the authors and reviewers share a vision of how to move the ball forward.   We’re also the umpires, making sure everyone is following the rules.  Conflict of interest?  Maybe.  But welcome to peer-review.

JFR has a three-part coaching strategy.

  • We’ll expand the playbook.  Gather data however you can; draw your theories from any discipline; define financial reporting broadly; draw conclusions from Bayesian analysis, machine learning, or qualitative interpretation.  Exploratory analyses, replications, null results—they’re all in our playbook.
  • We’ll settle for field goals.  Everyone’s looking for a touchdown. But it’s hard to explore new data sets and test theory; it’s hard to identify causality and generalize.  It’s hard to address deep issues and be useful right away. We’ll let you pursue a limited set of goals, as long as you do it well, and are honest about what you did and didn’t achieve.
  • We’ll go for two. OK, to make this metaphor to work, after a field goal you can get a couple more points by publishing discussions that run the ball over the goal line again, from right close up.   And what a difference it makes!  It’s not so much that we add commentary, but that we get to remove it from the paper itself.

Our goal is to accept or reject papers based on a narrow view of their substance: what did you do, and what did you find? We’ll hold substance to the highest standards.  But everything else is commentary, and reasonable disagreements shouldn’t be a bar to publication.  Instead, we’ll hash them out in discussions.

This gets at a lot of important issues in research communication.  I’m usually better at using communication channels than at designing methods for improving communication more generally, so I appreciate learning about work in this area.

Also, I like the football analogies.

17 thoughts on “Hey! Can you guess the 3 goofy tricks that this new journal is trying to improve peer review?

  1. I’m reminded about Microsoft’s approach to adopting new features (according to Raymond Chen):

    New features have to acquire a sufficiently high score to be accepted into the development process. Positive aspects or tradeoffs get positive points, negative ones get negative points.

    And … all new proposals start out with -100 points.

  2. One mistake–after a field goal, you don’t get to go for two.

    It’s a funny mistake because his analogy works well–we’ll settle for field goals, but when we do get a touchdown, we’ll take the chance and go for two

    • Yes, we needed to change the rules to allow 2 point conversions after a field goal. But that seemed trivial after the bigger change: the authors get to place their ball (the manuscript) on the field as far as they’ve moved it already on their own and with prior journals.

      p.s. Thanks Andrew for covering this. We need all the advice we can get! Much of the thinking behind this came from posts and comments on this site.

      • This objection is technically quite correct. There is no additional scoring opportunity after a field goal in American football.

        However, the point of the analogy is nonetheless well-taken.

        JFR should be commended for sticking its neck out there to take on these problems with traditional publications! The cynic in me foresees an uphill battle to deliver on that promise, though, _especially_ once the editor/editorial board changes. That goes double if the journal gets big/successful…

  3. Financial economists in Europe and Asia are frantically googling American football to figure out what a field goal is and whether they could publish one.

  4. “new editor…trying to make some changes to peer review”

    If a newish author submits a newish idea that challenges an existing paradigm, most journals will send it to peer reviewers who are vested in the paradigm. And those reviewers will be looking to magnify faults. That seems like an iron cage.

    “Second, we’ll base publication decisions on the substance of your work—what you actually do and find.”

    This makes it sound like the editors will be the primary reviewers. I get the sense that what you mean here is that if the peer reviewers find fault with the discussion parts of the paper, their objections will be given less weight by the editors than if they find fault with the substance of the paper. That would certainly be a good thing, but it is necessarily built on an expectation that authors will avoid pontificating too much. As an example:

    Amy Cuddy wrote:

    “That a person can, by assuming two simple 1-min poses, embody power and instantly become more powerful has real-world, actionable implications.”

    To which Andrew responded:

    “I bring up that sentence a lot because it’s just stunning, as the paper in question had no measures whatsoever of anyone “becoming more powerful.”

    Andrew was responding to a discussion point, not the substance of the paper. But you do want to weed out unsubstantiated claims.

    My own ruminations about peer review revolve around objective measures. Commenter Michael wrote on a previous thread:

    “If Slate can let Andrew Gelman write an article, or Retraction Watch can publish an interview with him expressing his position without publishing comments from experts with objectively equal qualifications who disagree, why can’t TED let Amy Cuddy put out her ideas?”

    If authority flows down from your qualifications, there is no good answer. But if it can be shown that Gelman’s claims are directly and robustly supported by his calculations and data, while Cuddy’s are not, Gelman’s claims are objectively better substantiated. The original idea of the p statistic was to impose a threshold on claims of correlation to weed out the worst ones, but authors learned to game it. It seems like we need a new generation of objective controls, and we need to impose them during peer review. I wish I knew what they were.

    • @Matt, you are right, JFR’s approach works only if we can change how reviewers evaluate papers and (as a result) how authors write them. Here’s some more detail on the guidance we provide reviewers:

      EVALUATION OF SUBSTANCE
      Reviewers should evaluate the substance of the paper-what the authors did, found and concluded-separate from commentary. Editors will use this evaluation to determine whether the paper is publishable. In particular, we ask authors to address the following questions about the paper’s substance:

      • Are the authors’ methods of modeling, data gathering, analysis and drawing conclusions appropriate to their question?
      • Are the manuscripts’ substantive conclusion supported by what they did and found?
      • Does the manuscript describe what was done, found and concluded in sufficient detail?
      • Do the authors clearly indicate the strengths and limitations of the work?
      • What particular choices by the authors require more justification?
      For papers that are ultimately rejected, a clear explanation of the limitations is essential so that authors can understand the basis for the decision. For papers judged as “revise and resubmit,” a good review will explain how the authors can overcome the limitations and strengthen the paper to the point that it is acceptable for publication.

      EVALUATION OF COMMENTARY

      Our goal is to publish manuscripts with strong substance and reasonable commentary. If reviewers believe the commentary is reasonable but arguably wrong, we allow them to write their own commentary (or suggest others to do so). We therefore ask reviewers to evaluate the commentary included in a manuscript by addressing the following questions:

      • Is commentary separated clearly from substance?
      • Are any of the claims in the manuscript’s commentary unreasonable (as opposed to reasonable but arguably wrong)?
      • How might you or others respond to commentary in the manuscript that is reasonable but arguably wrong?

  5. @RJB “In particular, we ask authors to address the following questions about the paper’s substance:”

    1) Do you mean “reviewers” rather than “authors”?

    2) Unfortunately, “address” is a very vague word, subject to a wide variety of interpretations. Guidelines need to be more explicit than that.

    I say this realizing full well that one can go overboard and give an overly specific set of criteria that then become just “hoops” to practice going through.

    What I have in mind would (among other things) incorporate the last bullet into each of the first three. e.g., (as just a first attempt at a partial improvement), the first bullet might be strengthened to read something like,

    “Have the authors given sound justifications of why their methods of modeling, data gathering, analysis and drawing conclusions are appropriate to their question? If not, what choices are not well justified? Are there reasons why any of their choices are inadequate or otherwise inappropriate? Or why other methods would be more appropriate and also feasible?”

    • Yep, that should be “we ask reviewers to…”

      Your questions about justification are worth thinking about. I mostly agree. However, manuscripts in our field tend to be very long (30 pages double-spaced, plus tables), and I think a lot of the length is unnecessary because people are rehashing justifications that have already been discussed extensively in the literature.

      I’ve been thinking it is worth distinguishing between asking whether methods are justifiable, and asking whether they are justified. The first means that they chose state-of-the-art methods that do the job they that needs to be done, and the second means that they explained why that is true. I have no problem asking reviewers to weigh in on whether methods are justifiable, but I am more reluctant to ask authors to justify everything, especially if they are borrowing heavily from prior work, which they typically are doing in our field.

      • If justification for using a particular method is given in prior work, then a specific reference to that justification could constitute justification in a new paper, so the argument that asking for justification would make papers unnecessarily long breaks down.

  6. I have nothing of substance to add, but IMHO there is something wrong with the title’s grammar. Or maybe with my grammar filter. But I will leave it out there for a discussion section.

    • It’s not the grammar, but unclear language. So it would be better to say, e.g.,”trying in order to improve” or “trying out to improve.”. (The title, as written, might be made clearer if spoken by intonation and pacing between “trying” and “to”.)

    • “Can you guess what Andrew is trying to annoy us?” That is like fingernails on a chalkboard, isn’t it?

      “Can you guess how Andrew is trying to annoy us?” seems much better. “Can you guess what techniques Andrew is using to try to annoy us?”

Leave a Reply to Tom Passin Cancel reply

Your email address will not be published. Required fields are marked *