Jeff points us to this news article by Asher Mullard:
Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility.
An unspoken industry rule alleges that at least 50% of published studies from academic laboratories cannot be repeated in an industrial setting, wrote venture capitalist Bruce Booth in a recent blog post. A first-of-a-kind analysis of Bayer’s internal efforts to validate ‘new drug target’ claims now not only supports this view but suggests that 50% may be an underestimate; the company’s in-house experimental data do not match literature claims in 65% of target-validation projects, leading to project discontinuation. . . .
Khusru Asadullah, Head of Target Discovery at Bayer, and his colleagues looked back at 67 target-validation projects, covering the majority of Bayer’s work in oncology, women’s health and cardiovascular medicine over the past 4 years. Of these, results from internal experiments matched up with the published findings in only 14 projects, but were highly inconsistent in 43 (in a further 10 projects, claims were rated as mostly reproducible, partially reproducible or not applicable . . .
High-impact journals did not seem to publish more robust claims, and, surprisingly, the confirmation of any given finding by another academic group did not improve data reliability. “We didn’t see that a target is more likely to be validated if it was reported in ten publications or in two publications,” says Asadullah. . . .
There’s also this amusing bit:
The analysis is limited by a small sample size, and cannot itself be checked because of company confidentiality concerns . . .
I suppose that the problem exists in political science as well, with journals only making the problem worse. Journals want big, significant effects and publish work that finds them. Scholars trying to “replicate” other papers are hoping for different findings. These are optimal conditions for making type M errors.
It would be interesting if journals would publish very short replication articles, with the criteria for acceptance revolving around how well the authors replicate the original paper’s procedure. This would encourage true replication studies, but also give the original authors an incentive to get it right the first time.
[...] 4:55 pmIn light of the recent article about drug-target research and replication (Andrew blogged it here) and l’affaire Potti, I have mentioned the “Forensic Bioinformatics” paper (Baggerly & [...]
[...] on Twitter and suggests that 50% of academic studies are completely wrong. Andrew Gelman briefly discusses it, and so do Tyler Cowen and Bruce [...]
[...] things off with a post about the statistical significance filter, and following up by commenting on type M errors in the lab and featuring a post about data management [...]