The “fish MRI” of international relations studies.

Kevin Lewis pointed me to this paper by Stephen Chaudoin, Jude Hays and Raymond Hicks, “Do We Really Know the WTO Cures Cancer?”, which begins:

This article uses a replication experiment of ninety-four specifications from sixteen different studies to show the severity of the problem of selection on unobservables. Using a variety of approaches, it shows that membership in the General Agreement on Tariffs and Trade/World Trade Organization has a significant effect on a surprisingly high number of dependent variables (34 per cent) that have little or no theoretical relationship to the WTO. To make the exercise even more conservative, the study demonstrates that membership in a low-impact environmental treaty, the Convention on Trade in Endangered Species, yields similarly high false positive rates. The authors advocate theoretically informed sensitivity analysis, showing how prior theoretical knowledge conditions the crucial choice of covariates for sensitivity tests. While the current study focuses on international institutions, the arguments also apply to other subfields and applications.

My reply: I’m not a fan of the “false positive” framework, but the general attitude expressed in the paper makes sense to me and I’m guessing this paper will be a very useful contribution to the literature in its field. It’s the “fish MRI” of international relations studies.

9 thoughts on “The “fish MRI” of international relations studies.

  1. membership in the General Agreement on Tariffs and Trade/World Trade Organization has a significant effect on a surprisingly high number of dependent variables (34 per cent) that have little or no theoretical relationship to the WTO

    Without reading the paper, this sounds surprisingly low. Either their sample size is too small or many of these measurements are close to random noise.

  2. This seems to be another confirmation of what Meehl said all those years ago: in a complex system, there will be “significant” correlations between many or even most of the variables you choose to look at.

  3. “We calculated each p-value in the same way that the authors did, e.g. robust or clustered standard errors. We are interested in the likelihood that selection effects cause incorrect inferences, as opposed to the possibility that incorrect statistical calculations cause incorrect inferences. For work on the latter subject, see Bertrand, Duflo, and Mullainathan.”

    …yeah but how can they disentangle the two? I mean, I don’t see any “placebo” type confirmation where they show that the SE estimators they use generate good coverage rates in the absence of selection*. Maybe I missed it, but I just don’t know how we can separate the two without showing that, given the empirical setup we have, if WTO membership were randomly assigned then we would get back our 0.05 rejection rates.

    Also, I think the BDM paper referenced is closer to the “Dead Fish” paper than this one… in BDM and Dead Fish (I think), the inference calculations are wrong. In this case, the authors are arguing that the point-estimate calculations are wrong (biased) and the inference calculations are fine in the sense of the confidence intervals having the appropriate width, just centered in the wrong place.

Leave a Reply to dmk38 Cancel reply

Your email address will not be published. Required fields are marked *