Skip to content
Archive of posts filed under the Causal Inference category.

Problems with “incremental validity” or more generally in interpreting more than one regression coefficient at a time

Kevin Lewis points us to this interesting paper by Jacob Westfall and Tal Yarkoni entitled, “Statistically Controlling for Confounding Constructs Is Harder than You Think.” Westfall and Yarkoni write: A common goal of statistical analysis in the social sciences is to draw inferences about the relative contributions of different variables to some outcome variable. When […]

Problems with randomized controlled trials (or any bounded statistical analysis) and thinking more seriously about story time

In 2010, I wrote: As a statistician, I was trained to think of randomized experimentation as representing the gold standard of knowledge in the social sciences, and, despite having seen occasional arguments to the contrary, I still hold that view, expressed pithily by Box, Hunter, and Hunter (1978) that “To find out what happens when […]

Field Experiments and Their Critics

Seven years ago I was contacted by Dawn Teele, who was then a graduate student and is now a professor of political science, and asked for my comments on an edited book she was preparing on social science experiments and their critics. I responded as follows: This is a great idea for a project. My […]

About that claim in the Monkey Cage that North Korea had “moderate” electoral integrity . . .

Yesterday I wrote about problems with the Electoral Integrity Project, a set of expert surveys that are intended to “evaluate the state of the world’s elections” but have some problems, notably rating more than half of the U.S. states in 2016 as having lower integrity than Cuba (!) and North Korea (!!!) in 2014. I […]

Transformative treatments

Kieran Healy and Laurie Paul wrote a new article, “Transformative Treatments,” (see also here) which reminds me a bit of my article with Guido, “Why ask why? Forward causal inference and reverse causal questions.” Healy and Paul’s article begins: Contemporary social-scientific research seeks to identify specific causal mechanisms for outcomes of theoretical interest. Experiments that […]

Sorry, but no, you can’t learn causality by looking at the third moment of regression residuals

Under the subject line “Legit?”, Kevin Lewis pointed me to this press release, “New statistical approach will help researchers better determine cause-effect.” I responded, “No link to any of the research papers, so cannot evaluate.” In writing this post I thought I’d go further. The press release mentions 6 published articles so I googled the […]

You’ll have to figure this one out for yourselves.

So. The other day this following email comes in, subject line “Grabbing headlines using poor statistical methods,” from Clifford Anderson-Bergman:

How can time series information be used to choose a control group?

This post is by Phil Price, not Andrew. Before I get to my question, you need some background. The amount of electricity that is provided by an electric utility at a given time is called the “electric load”, and the time series of electric load is called the “load shape.” Figure 1 (which is labeled […]

OK, sometimes the concept of “false positive” makes sense.

Paul Alper writes: I know by searching your blog that you hold the position, “I’m negative on the expression ‘false positives.’” Nevertheless, I came across this. In the medical/police/judicial world, false positive is a very serious issue: $2 Cost of a typical roadside drug test kit used by police departments. Namely, is that white powder […]

How effective (or counterproductive) is universal child care? Part 2

This is the second of a series of two posts. Yesterday we discussed the difficulties of learning from a small, noisy experiment, in the context of a longitudinal study conducted in Jamaica where researchers reported that an early-childhood intervention program caused a 42%, or 25%, gain in later earnings. I expressed skepticism. Today I want […]

How effective (or counterproductive) is universal child care? Part 1

This is the first of a series of two posts. We’ve talked before about various empirically-based claims of the effectiveness of early childhood intervention. In a much-publicized 2013 paper based on a study of 130 four-year-old children in Jamaica, Paul Gertler et al. claimed that a particular program caused a 42% increase in the participants’ […]

A four-way conversation on weighting and regression for causal inference

It started with a project that Sharad Goel is doing, comparing decisions of judges in an urban court system. Sharad was talking with Avi Feller, Art Owen, and me about estimating the effect of a certain decision option that judges have, controlling for pre-treatment differences between defendants. Art: I’m interested in what that data shows […]

Take that, Bruno Frey! Pharma company busts through Arrow’s theorem, sets new record!

I will tell a story and then ask a question. The story: “Thousands of Americans are alive today because they were luckily selected to be in the placebo arm of the study” Paul Alper writes: As far as I can tell, you have never written about Tambocor (Flecainide) and the so-called CAST study. A locally […]

Birthdays and heat waves

I mentioned the birthdays example in a talk the other day, and Hal Varian pointed me to some research by David Lam and Jeffrey Miron, papers from the 1990s with titles like Seasonality of Births in Human Populations, The Effect of Temperature on Human Fertility, and Modeling Seasonality in Fecundability, Conceptions, and Births. Aki and […]

Balancing bias and variance in the design of behavioral studies: The importance of careful measurement in randomized experiments

At Bank Underground: When studying the effects of interventions on individual behavior, the experimental research template is typically: Gather a bunch of people who are willing to participate in an experiment, randomly divide them into two groups, assign one treatment to group A and the other to group B, then measure the outcomes. If you […]

Calorie labeling reduces obesity Obesity increased more slowly in California, Seattle, Portland (Oregon), and NYC, compared to some other places in the west coast and northeast that didn’t have calorie labeling

Ted Kyle writes: I wonder if you might have some perspective to offer on this analysis by Partha Deb and Carmen Vargas regarding restaurant calorie counts. [Thin columnist] Cass Sunstein says it proves “that calorie labels have had a large and beneficial effect on those who most need them.” I wonder about the impact of […]

Does Benadryl make you senile? Challenges in research communication

Mark Tuttle points to a post, “Common anticholinergic drugs like Benadryl linked to increased dementia risk” by Beverly Merz, Executive Editor, Harvard Women’s Health Watch. Merz writes: In a report published in JAMA Internal Medicine, researchers offers compelling evidence of a link between long-term use of anticholinergic medications like Benadryl and dementia. . . . […]

Killer O

Taggert Brooks points to this excellent news article by George Johnson, who reports: Epidemiologists have long been puzzled by a strange pattern in their data: People living at higher altitudes appear less likely to get lung cancer. . . . The higher you live, the thinner the air, so maybe oxygen is a cause of […]

One-day workshop on causal inference (NYC, Sat. 16 July)

James Savage is teaching a one-day workshop on causal inference this coming Saturday (16 July) in New York using RStanArm. Here’s a link to the details: One-day workshop on causal inference Here’s the course outline: How do prices affect sales? What is the uplift from a marketing decision? By how much will studying for an […]

About that claim that police are less likely to shoot blacks than whites

Josh Miller writes: Did you see this splashy NYT headline, “Surprising New Evidence Shows Bias in Police Use of Force but Not in Shootings”? It’s actually looks like a cool study overall, with granular data, and a ton of leg work, and rich set of results that extend beyond the attention grabbing headline that is […]