Skip to content
Archive of posts filed under the Causal Inference category.

About that quasi-retracted study on the Mediterranean diet . . .

Some people asked me what I thought about this story. A reporter wrote to me about it last week, asking if it looked like fraud. Here’s my reply: Based on the description, there does not seem to be the implication of fraud. The editor’s report mentioned “protocol deviations, including the enrollment of participants who were […]

Some experiments are just too noisy to tell us much of anything at all: Political science edition

Sointu Leikas pointed us to this published research article, “Exposure to inequality affects support for redistribution.” Leikas writes that “it seems to be a really apt example of “researcher degrees of freedom.’” Here’s the abstract of the paper: As the world’s population grows more urban, encounters between members of different socioeconomic groups occur with greater […]

How to reduce Type M errors in exploratory research?

Miao Yu writes: Recently, I found this piece [a news article by Janet Pelley, Sulfur dioxide pollution tied to degraded sperm quality, published in Chemical & Engineering News] and the original paper [Inverse Association between Ambient Sulfur Dioxide Exposure and Semen Quality in Wuhan, China, by Yuewei Liu, published in Environmental Science & Technology]. Air […]

Does “status threat” explain the 2016 presidential vote?

Steve Morgan writes: The April 2018 article of Diana Mutz, Status Threat, Not Economic Hardship, Explains the 2016 Presidential Vote, was published in the Proceedings of the National Academy of Sciences and contradicts prior sociological research on the 2016 election. Mutz’s article received widespread media coverage because of the strength of its primary conclusion, declaimed […]

“16 and Pregnant”

Ted Joyce writes: In December 2015 the AER published an article, “Media Influences on Social Outcomes: The Impact of MTV’s 16 and Pregnant on Teen Childbearing,” by Melissa Kearney and Phil Levine [KL]. The NBER working paper of this article appeared in January of 2014. It received huge media attention as the authors claimed the […]

Early p-hacking investments substantially boost adult publication record

In a post with the title “Overstated findings, published in Science, on long-term health effects of a well-known early childhood program,” Perry Wilson writes: In this paper [“Early Childhood Investments Substantially Boost Adult Health,” by Frances Campbell, Gabriella Conti, James Heckman, Seong Hyeok Moon, Rodrigo Pinto, Elizabeth Pungello, and Yi Pan], published in Science in […]

“The Internal and External Validity of the Regression Discontinuity Design: A Meta-Analysis of 15 Within-Study-Comparisons”

Jag Bhalla points to this post by Alex Tabarrok pointing to this paper, “The Internal and External Validity of the Regression Discontinuity Design: A Meta-Analysis of 15 Within-Study-Comparisons,” by Duncan Chaplin, Thomas Cook, Jelena Zurovac, Jared Coopersmith, Mariel Finucane, Lauren Vollmer, and Rebecca Morris, which reports that regression discontinuity (RD) estimation performed well in these […]

Does adding women to corporate boards increase stock price?

Anton Kasster writes:

Bayesian inference for A/B testing: Lauren Kennedy and I speak at the NYC Women in Machine Learning and Data Science meetup tomorrow (Tues 27 Mar) 7pm

Here it is: Bayesian inference for A/B testing Andrew Gelman, Department of Statistics and Department of Political Science, Columbia University Lauren Kennedy, Columbia Population Research Center, Columbia University Suppose we want to use empirical data to compare two or more decisions or treatment options. Classical statistical methods based on statistical significance and p-values break down […]

Debate over claims of importance of spending on Obamacare advertising

Jerrod Anderson points to this post by Paul Shafer, Erika Fowler, Laura Baum, and Sarah Gollust, “Advertising cutbacks reduce Marketplace information-seeking behavior: Lessons from Kentucky for 2018.” Anderson expresses skepticism about this claim. I’ll first summarize the claims of Shafer et al. and then get to Anderson’s criticism. Shafer et al. write: The Trump administration […]

The moral hazard of quantitative social science: Causal identification, statistical inference, and policy

A couple people pointed me to this article, “The Moral Hazard of Lifesaving Innovations: Naloxone Access, Opioid Abuse, and Crime,” by Jennifer Doleac and Anita Mukherjee, which begins: The United States is experiencing an epidemic of opioid abuse. In response, many states have increased access to Naloxone, a drug that can save lives when administered […]

“and, indeed, that my study is consistent with X having a negative effect on Y.”

David Allison shares this article: Pediatrics: letter to the editor – Metformin for Obesity in Prepubertal and Pubertal Children A Randomized Controlled Trial and the authors’ reply: RE: Clarification of statistical interpretation in metformin trial paper The authors of the original paper were polite in their response, but they didn’t seem to get the point […]

Bayes for estimating a small effect in the context of large variation

Shira Mitchell and Mariel Finucane, two statisticians at Mathematica Policy Research (that’s the policy-analysis organization, not the Wolfram software company) write: We here at Mathematica have questions about priors for a health policy evaluation. Here’s the setting: In our dataset, healthcare (per person per month) expenditures are highly variable (sd = $2500), but from prior […]

I’ll use this line in my talk this Wednesday at the Society for Research on Educational Effectivness

I had a conversation with a policy analyst about the design of studies for program evaluation—the post is scheduled to appear in a few months—and he expressed some frustration: The idea of evidence based policy has put a gun to our heads as researchers to give binary responses with absolute confidence to a question that […]

One data pattern, many interpretations

David Pittelli points us to this paper: “When Is Higher Neuroticism Protective Against Death? Findings From UK Biobank,” and writes: They come to a rather absurd conclusion, in my opinion, which is that neuroticism is protective if, and only if, you say you are in bad health, overlooking the probability that neuroticism instead makes you […]

Testing Seth Roberts’ appetite theory

Jonathan Tupper writes: My organization is running a group test of Seth Roberts’ old theory about appetite. We are running something like a “web trial” as discussed in your Chance article with Seth. And in fact our design was very inspired by your conversation… For one, we are using a control group which takes light […]

I’m skeptical of the claims made in this paper

Two different people pointed me to a recent research article, suggesting that the claims therein were implausible and the result of some combination of forking paths and spurious correlations—that is, there was doubt that the results would show up in a preregistered replication, and that, if they did show up, that they would mean what […]

What’s Wrong with “Evidence-Based Medicine” and How Can We Do Better? (My talk at the University of Michigan Friday 2pm)

Tomorrow (Fri 9 Feb) 2pm at the NCRC Research Auditorium (Building 10) at the University of Michigan: What’s Wrong with “Evidence-Based Medicine” and How Can We Do Better? Andrew Gelman, Department of Statistics and Department of Political Science, Columbia University “Evidence-based medicine” sounds like a good idea, but it can run into problems when the […]

354 possible control groups; what to do?

Jonas Cederlöf writes: I’m a PhD student in economics at Stockholm University and a frequent reader of your blog. I have for a long time followed your quest in trying to bring attention to p-hacking and multiple comparison problems in research. I’m now myself faced with the aforementioned problem and want to at the very […]

The difference between me and you is that I’m not on fire

“Eat what you are while you’re falling apart and it opened a can of worms. The gun’s in my hand and I know it looks bad, but believe me I’m innocent.” – Mclusky While the next episode of Madam Secretary buffers on terrible hotel internet, I (the other other white meat) thought I’d pop in […]