Skip to content
 

When do we want evidence-based change? Not “after peer review”

Jonathan Falk sent me the above image in an email with subject line, “If this isn’t the picture for some future blog entry I’ll never forgive you.” This was a credible threat so here’s the post.

But I don’t agree with that placard at all!

Waiting for peer review is a bad idea for two reasons: First because you might be waiting for a really long time (especially if an econ journal is involved), second because all sorts of bad stuff gets through peer review—just search this blog for PPNAS.

Falk replied:

Completely agree… That’s what makes it funny, no?

Making a living outside academia is all about this for me. When asked on the stand whether some analysis of mine has had peer review, my answer is yes, because my colleagues have reviewed it. But if we had to defend on traditional academic peer review, the time lags would put us out of business—buy of course the REAL peer review is performed by the experts in the other side. Consulting experts are the replication ideal, no?

I don’t know enough about legal consulting to have an opinion on whether consulting experts are the replication ideal, but I do know that I don’t want to wait around for peer review.

Another way to see this is from a decision-analytic perspective. There are potential costs and benefits to change, potential costs and benefits to remaining with the status quo, and potential costs and benefits to waiting for more information. A decision among these options has to depend to some extent on these costs and benefits, which are considered only obliquely in any peer review process.

16 Comments

  1. Dale Lehman says:

    I have often held up regulatory proceedings as a superior method of review for empirical work – cross examination and expert testimony from various stakeholders seems to work better than peer review in journals, for a variety of reasons. I maintain that position. But, I would not go so far as to claim this is the “replication ideal.” Not even close. For that ideal to be realized, the regulators, judges, etc. would need to do a much better job of really wanting to evaluate the merits of the evidence. After years of being an expert witness, I would not characterize the legal process as working like that. It more closely resembles a well-orchestrated (or not so well, in many cases) Kabuki theater, where decisions have been made for political and personal reasons. Often cross-examination is planned to avoid achieving clarity about the evidence and the analysis that was conducted. It is a rare case where competing positions are articulated and critiqued.

    If regulators and judges wanted to achieve the “replication ideal,” they could easily do so. They only need announce that testimony that does not provide the data and methodology will be ignored, they would use a panel format so that “experts” must engage with each other, and they would employ their own (relatively unbiased) experts to advise them on technical matters. That would be an ideal, but it is rarely (dare I say never) approached in practice.

    • Jonathan (another one) says:

      OK… so “replication ideal” was a bit strong. But I was younger when I wrote that, thanks to the blog delay. (I’d forgotten all about it!)

      But what I meant was that the other side actually does an actual replication, and reports on infirmities found and carries out their own research which (presumably) reaches the opposite conclusion, which is then replicated by the original researchers, etc.

      You bring out the undeniably correct (and frustrating) fact that what decisionmakers *do* with all this work is… ok, let’s just say opaque. And that is at least partially because the parties are (understandably) more interested in winning than in learning, and have a stilted (and generally uninformative) way of exploring these issues live.

      And I have been employed by a Court to help them interpret econometric evidence presented by the two warring sides. So it does happen, but not (I agree) nearly often enough.

  2. Matt Skaggs says:

    Welp. I don’t think the placard has anything to do with whether a different form of review is superior to peer review, or even when change should occur. Or maybe everything in the blog post is snark.

    I suspect the placard represents one of the many ironies of the climate wars. I interpret it to mean that we want society to change based upon the current “evidence-based” climate change paradigm, as described by those who do not need to worry about peer-review gatekeeping because they support that paradigm. Not taking a side here, just saying.

    The convoluted message meant by the placard would make an interesting blog post subject in itself!

    • Andrew says:

      Matt:

      My post is completely serious. No snark at all. I’m disagreeing with the statement that we want evidence-based change “after peer review.” As I wrote, I think this is a bad idea for two reasons: first, some evidence-based changes are important enough that they should not wait for peer review (at least, not if “peer review” is taken to mean the traditional version conducted by academic journals); second, lots of ideas that succeed with traditional peer review are bad ideas and I don’t think we should be making changes based on them.

      I you take the term “peer review” more generally, then I guess I do pretty much agree with the placard. If we’re going to make changes, we should consult with our peers first rather than acting unilaterally. But my impression was that the placard was not merely making that unobjectionable point; it was my impression that the placard was defending the traditional peer review process of established science.

      • Matt Skaggs says:

        Andrew:

        I did understand that your comments on scientific methodology were made in earnest. What I was not sure about was whether you (and Jonathan) were aware of what the placard holder was actually advocating, because the slogan on the placard is entirely orthogonal to the discussion here.

        I’m pretty sure that I recognize the placard from the Hockey Stick Wars, a skirmish within the larger Climate Wars. There is even a video of folks chanting this slogan at a climate rally! For students of science history, this may be the only time that political activists sang the praises of peer review at a political rally. But I digress.

        Here is the long, literalized version of the placard:

        What do we want?

        Reductions in CO2 emissions based upon evidence of human-caused global warming contained in scientific papers!

        How do we choose what evidence to believe?

        By whether the paper is peer reviewed!

        There is enough back story to fill a book, but let’s see if I can summarize down to a few sentences. The Hockey Stick Wars were about dodgy statistical treatments (perhaps discussed at this blog, I have only been reading a couple years). Contrary papers were written by skeptics, but the skeptical papers were not making it through peer review. (We know now that at least some of that was gatekeeping.) In response, some skeptics chose to have their papers published by entities that were entirely outside of academia and the journal publishing process, such as the Global Warming Policy Foundation. Those papers did not go through a peer review process, but some did get attention from the press. Some prominent climate scientists who had embraced the global warming consensus decided that the skeptical papers needed to be rebutted, but the points of dispute were highly technical and so not suitable for activist arguments. This led the scientists to suggest a razor, telling the activists to advocate an approach in which any paper that was not peer reviewed should be ignored. The placard slogan is a rather odd attempt by a political activist to capture the approach advocated by the climate scientists.

        There are rich layers of irony here:

        1. The statistical treatments proposed by the skeptics were often better than those used by the consensus scientists.
        2. Peer review was often pal review. There is objective evidence that it was very sloppy on many of the consensus papers.

        …and this one that actually touches on the discussion here:

        3. The climate science Annual Reviews (as published by the IPCC) had hard deadlines for paper submittal. There were consensus papers that were written specifically for inclusion in the Annual Reviews, but the papers had gotten bogged down in peer review. Those papers were cited anyway, meaning that policy was influenced by – yep, that’s right folks – papers that were technically not peer reviewed.

  3. Ben Prytherch says:

    I suspect that many (most?) science cheerleaders who talk about peer review don’t know how it works. I certainly didn’t before I came to academia. I just assumed it meant that you shared your results / argument / theory openly and then this was subjected to criticism by your peers. Because after all science is self correcting, and that’s how a self correcting institution should be set up. Hardy har har.

    The person holding this sign might just know of “peer review” as this thing that’s supposed to make science more reliable than non-science.

  4. I thought that Peer Review Congress proceeding in Chicago two weeks ago was helpful in thinking about Peer Review changes that need to take place.

  5. Tom Dietterich says:

    Gee, have a sense of humor Andrew. There were many of these signs as part of the March for Science. The point was to draw a contrast with the fake news anti-science crowd and also to make fun of the “When do we want it? Now!” chant of typical demonstrations. “Peer review” was simply a place-holder for some systematic method of evaluation of the scientific claims. A method that did not impulsively jump to take action but paused to assess the evidence first.

    The scientific enterprise is under attack not only from ambitious scientists trying to sell false certainty and false significance but also from folks deliberately trying to sow fear, uncertainty, and doubt. While this blog focuses on the former, the danger of the latter may be the greater.

    • Anonymous says:

      “The scientific enterprise is under attack not only from ambitious scientists trying to sell false certainty and false significance but also from folks deliberately trying to sow fear, uncertainty, and doubt. While this blog focuses on the former, the danger of the latter may be the greater.”

      I think uncertainty and doubt in science can only be a good thing, and i actually think it could be “dangerous” to not keep this in mind when reading anything science-related.

      To give a concrete example, it reminded me of some information about dietary guidelines i recently came across in my search for information about “optimizing” my diet. The general population may have been massively misinformed, for decades, about eating carbs and fat. If correct, it makes me wonder how much damage this has done to the general public…

      http://time.com/4130043/lobbying-politics-dietary-guidelines/

      “Throughout the 1980s and 1990s, the USDA treated fat as the primary harm in the American diet,” says Nestle. Along with its anti-fat stance—a stance researchers say was never grounded in science—the guidelines also encouraged Americans to eat hefty amounts of carbohydrates. The 1995 edition made bread, cereal, and pasta the foundation of its “Food Guide Pyramid,” and advised people to eat between six and 11 servings of grains every day, compared to just three to five servings of vegetables and two to four servings of fruit. Fat was to be eaten “sparingly.”

      “This advice to eat more carbs and avoid fat is exactly backwards if you want to improve health and lower body weight,” says Dr. Robert Lustig, a pediatric endocrinologist at the University of California, San Francisco. He and other nutrition researchers say the popularity of anti-fat, pro-carb guidelines helped fuel a rise in diet-related health problems.

      Since 1980, when the government issued its first set of dietary guidelines, the number of Americans who are obese or have type-2 diabetes has more than doubled. Roughly half of all American adults now live with one or more chronic, preventable diseases, and rates of childhood obesity have reached “epidemic” proportions, according to the National Institutes of Health.”

      I am going to make breakfast now. No bread for me, but (what i hope are) healthy fats: avocado, bacon strips, and an omelette :)

  6. Shravan says:

    For me, as a psycholinguist, peer review has almost always improved my papers. The only times it didn’t work were when reviewers used their power to get ahead by preventing (or trying to prevent) publications that went against their published claim (pdycholx is a great place to study Type M errors in the wild), when they had through a random process become an expert without knowing anything (“what is the use of replicating a published result”?, “please demonstrate lower p-values to establish replicability”), or wanted to publish their own paper in a top journal but didn’t want our results to appear in that journal. These cases of misconduct were pretty rare though. Most people review constructively. Some did damage my students’ careers by holding the paper in the editor’s inbox, without review, for 9 months.

    • Andrew says:

      Shravan:

      I agree. Peer review can be great, and it has almost always improved my papers too, sometimes in very important ways. I also think that sometimes the threat of peer review has made my papers worse, in that I’ve either not published a result because I anticipated it would not be considered publishable by reviewers, or I’ve written a paper in an awkward way in anticipation of reviewers’ attitudes.

      • shravan says:

        Yes, i also have experienced the situation that i know how reviewers will react so i end up writing a different paper than i would have written if unconstrained. Right now i am writing a book on sentence processing, and realized, shocked, that nobody can stop me from saying what i think. I already have the book contract! I could get used to this freedom.

      • Keith O'Rourke says:

        The scent of selective publication serpents is everywhere :-(

Leave a Reply