Type M errors in the lab

Jeff points us to this news article by Asher Mullard:

Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility.

An unspoken industry rule alleges that at least 50% of published studies from academic laboratories cannot be repeated in an industrial setting, wrote venture capitalist Bruce Booth in a recent blog post. A first-of-a-kind analysis of Bayer’s internal efforts to validate ‘new drug target’ claims now not only supports this view but suggests that 50% may be an underestimate; the company’s in-house experimental data do not match literature claims in 65% of target-validation projects, leading to project discontinuation. . . .

Khusru Asadullah, Head of Target Discovery at Bayer, and his colleagues looked back at 67 target-validation projects, covering the majority of Bayer’s work in oncology, women’s health and cardiovascular medicine over the past 4 years. Of these, results from internal experiments matched up with the published findings in only 14 projects, but were highly inconsistent in 43 (in a further 10 projects, claims were rated as mostly reproducible, partially reproducible or not applicable . . .

High-impact journals did not seem to publish more robust claims, and, surprisingly, the confirmation of any given finding by another academic group did not improve data reliability. “We didn’t see that a target is more likely to be validated if it was reported in ten publications or in two publications,” says Asadullah. . . .

There’s also this amusing bit:

The analysis is limited by a small sample size, and cannot itself be checked because of company confidentiality concerns . . .

More here and here.

4 thoughts on “Type M errors in the lab

  1. I suppose that the problem exists in political science as well, with journals only making the problem worse. Journals want big, significant effects and publish work that finds them. Scholars trying to “replicate” other papers are hoping for different findings. These are optimal conditions for making type M errors.

    It would be interesting if journals would publish very short replication articles, with the criteria for acceptance revolving around how well the authors replicate the original paper’s procedure. This would encourage true replication studies, but also give the original authors an incentive to get it right the first time.

  2. Pingback: Reproducibility in Practice « Statistical Modeling, Causal Inference, and Social Science

  3. Pingback: A Structure to Encourage Reproducibility | Bootstrapping Life

  4. Pingback: Pie Charts and Maps, Reproducibility, and Social versus Physical Sciences | Carlisle Rainey

Comments are closed.