Via Yalda Afshar, a 2005 paper by Hans-Hermann Dubben and Hans-Peter Beck-Bornholdt:
Publication bias is a well known phenomenon in clinical literature, in which positive results have a better chance of being published, are published earlier, and are published in journals with higher impact factors. Conclusions exclusively based on published studies, therefore, can be misleading. Selective under-reporting of research might be more widespread and more likely to have adverse consequences for patients than publication of deliberately falsified data. We investigated whether there is preferential publication of positive papers on publication bias.
They conclude, “We found no evidence of publication bias in reports on publication bias.” But of course that’s the sort of finding regarding publication bias of findings on publication bias that you’d expect would get published. What we really need is a careful meta-analysis to estimate the level of publication bias in studies of publication bias of studies of publication bias.
Gödel strikes again!
What? Wasn’t that a negative result, not a positive one? The joke must have seemed obligatory but it doesn’t really work.
[...] That is from MR reader Kevin Burke, who refers me to this post from the excellent Andrew Gelman. [...]
Sam, finding a publication bias in reports on publication bias might have called the findings in this report on publication bias in reports on publication bias into question, leading to it not getting published.
I agree with Sam. This is a fun paper to read for all the sentences that make your head spin, but I think we should count ourselves lucky. A positive result might have just broken statistics.
In this TED talk, Ben Goldacre shows a funnel plot giving apparently strong evidence for publication bias in studies of publication bias in pharmaceutical trials. He calls it ‘the funniest epidemiology joke that you will ever hear’ (approx. 11 mins into the talk). I don’t know the source for the graph, though.
I think the graph Ben Goldacre shows is from the paper that’s linked. The p-value is 0.13,but the graph certainly looks like bias.
Andrew : also maybe you are right about the NULL hypothesis not making sense (here).
They may be no (accessible) literature where there is not (noticeable) publication bias!
Eli: The above possibility suggests the opposite to “A positive result might have just broken statistics.”
Tom: Nice TED talk, thanks.
But, if I heard them right the miss-interpreted the funnel plot as providing evidence for publication bias. I have always been very wary of funnel plots as publication decisions are highly multivariate, non-linear and non-common (over journals and time). I saw some promise in John Copas and students work on modelling publication bias less wrongly, but Greenland seemed far less hopeful.
But as the authors point out, power was likely an issue and without addressing what the power likely was (based on an informative prior on “effect” size) there has been many arguments given on this blog that you should not try to make anything of the lack of statistical significance in the results….
We just need less wrong models of less wrong models of less wrong models…
Would it be possible to make a paper that reports its own bias? Would that close the loop?
who watches the watchers of the watch men? An author of a paper on the publication biases in publications about publication biases has good reason to not find publication bias… you already know he’s the kind of person who wants to affiliate with the ‘good guys.’
[...] clinical results tend to be published less often than positive ones. Apparently, there’s no publication bias in studies that examine publication bias. Try wrapping your head around [...]
A quick question:
Is there an analogy to publication bias in terms of citation? This is, “citation bias”.
For example: citing only your friends, your crush or people working on the same school of thought and keeping, consciously, other possible references under the carpet.