The Night Riders

Retraction Watch linked to this paper, “Publication bias and the canonization of false facts,” by Silas Nissen, Tali Magidson, Kevin Gross, and Carl Bergstrom, and which is in the Physics and Society section of Arxiv which is kind of odd since it has nothing whatsoever to do with physics. Nissen et al. write:

In the process of scientific inquiry, certain claims accumulate enough support to be established as facts. Unfortunately, not every claim accorded the status of fact turns out to be true. In this paper, we model the dynamic process by which claims are canonized as fact through repeated experimental confirmation. . . . In our model, publication bias—in which positive results are published preferentially over negative ones—influences the distribution of published results.

I don’t really have any comments on the paper itself—I’m never sure when these mathematical models are adding to our understanding of social processes, and when they’re just adding confusion—but there was a side point to be made, and this is that there’s selection bias within, as well as between, publications. Indeed, I suspect the “within” bias is larger than the “between.”

Consider two scenarios:

1. Within-publication bias: A researcher studies topic X, gathers data, does what it takes to get a publication. There’s a bias toward finding statistical significance, a bias toward overestimating effect sizes, a bias toward overconfidence in conclusions, and a bias toward finding something ideologically appealing to the researcher.

2. Between-publication bias: A researcher, or set of researchers, work on a series of projects. Some are successful and some are not. The successful studies get published. This results in the same biases as before.

Both scenarios happen, but I suspect that scenario 1, within-publication bias, is more important. I don’t think researchers have so many papers in their file drawers, and I also think that they have lots of degrees of freedom allowing them to find success in their data.

I wrote the above post because I worry that when people talk about publication bias, they’re thinking too much about the publication/nonpublication decision, and not enough about all the bias that goes into what people decide to report at all.

5 thoughts on “The Night Riders

  1. First, the authors seem to address the within-paper bias:

    “This problem is exacerbated when scientists engage in p-hacking, data dredging, and other behaviors that increase the rate at which false positives are published.”

    See section E and Fig 8.

    Second, mathematical models have their own form of harking, in that the conclusions follow from the assumptions. I’ve seen plenty of mathematical models whose results were so easily foreordained by the assumptions that turning the mathematical cranks added little beyond the assumptions, and, in my view, was an illegitimate way to add mathematical authority to those assumptions rather than the conclusions that follow from them (not saying that is true of this paper, however).

  2. “I wrote the above post because I worry that when people talk about publication bias, they’re thinking too much about the publication/nonpublication decision, and not enough about all the bias that goes into what people decide to report at all.”

    Hmm, i am sorry but i don’t agree. Or perhaps better phrased, a) i think there already are appropriate terms to describe the things in scenario 1, b) i don’t understand scenario 2, and most importantly c) i don’t think it’s useful to introduce new terms if they don’t add anything useful to the discussion.

    Scenario 1 to me can be described using different terms that are already being used, such as selective reporting of analyses, p-hacking, garden of forking paths, etc. If you want to use a single term for all these things, perhaps you could call it “research(er) bias”.

    I agree that “they’re thinking too much about the publication/non-publication decision” in the sense that i think most researchers view that the publication/non-publication is the essential part of publication bias. I think this is correct and useful (which is why i would leave out the “too much”). It also makes sense to me given the term “publication” bias, i.c. it’s about publication.

    I came across several definition/descriptions of “publication bias” a while ago when trying to find a definition to use. Lots of these put the emphasis on only *significant* findings being published. I think this is wrong, or sub-optimal at best. Although that may have been the case thus far, you can perhaps also claim that in the past “successful” replications may have not found there way in the journals because they were not “new” or “innovative” enough.

    I actually like the following definition from wikipedia:

    “Publication bias is a type of bias that occurs in published academic research. It occurs when the outcome of an experiment or research study influences the decision whether to publish or otherwise distribute it.”

    (I reason that this definition incorporates several scenarios. When both significant or non-significant findings are being withheld, and could also involve scenarios where the reason for possible withholding has more to do with the author, topic, research, or conclusions.)

    Perhaps “research(er) bias” could be defined as:

    “Research bias is a type of bias that occurs in performing academic research. It occurs when the desired outcome of an experiment or research study influences the decisions regarding experimental set-up, referred to/used literature, data-analyses, data- (re-) presentation, conclusions drawn, etc.”

    (I reason that this definition incorporates several scenarios. When both significant or non-significant findings are being seen as “desired”, and could also involve scenarios where the “desired” outcome has more to do with the author, topic, research, or conclusions.)

  3. Isn’t there some overlap in the issues of forking paths and file-drawers?

    For example, if a researcher were to publish all their forking-paths that they examined before they landed on their “discovery”, this would make a meta-analysis that included all the earlier effects examined much less biased.

    Now, I will state this is only in theory rather than practice. As you have stated, forking-paths are not often not realized by the researcher. Similarly, the lack of forking paths would be part of the same issue; if researcher A sees a somewhat quadratic trend in their data, but researcher B does not, then the literature has A showing evidence for a quadratic effect and B not examining for a quadratic trend.

    Open data is the only way to go. Fork your paths and check on new data!

Leave a Reply to Ed Hagen Cancel reply

Your email address will not be published. Required fields are marked *