Peter Bergman points me to this discussion from Cyrus of a presentation by Guido Imbens on design of randomized experiments. Cyrus writes: The standard analysis that Imbens proposes includes (1) a Fisher-type permutation test of the sharp null hypothesis–what Imbens referred to as “testing”–along with a (2) Neyman-type point estimate of the sample average treatment [...]
Archive of posts filed under the Causal Inference category.
Macro causality
David Backus writes: This is from my area of work, macroeconomics. The suggestion here is that the economy is growing slowly because consumers aren’t spending money. But how do we know it’s not the reverse: that consumers are spending less because the economy isn’t doing well. As a teacher, I can tell you that it’s [...]
Matching and regression: two great tastes etc etc
Matthew Bogard writes: Regarding the book Mostly Harmless Econometrics, you state: A casual reader of the book might be left with the unfortunate impression that matching is a competitor to regression rather than a tool for making regression more effective. But in fact isn’t that what they are arguing, that, in a ‘mostly harmless way’ [...]
Descriptive statistics, causal inference, and story time
Dave Backus points me to this review by anthropologist Mike McGovern of two books by economist Paul Collier on the politics of economic development in Africa. My first reaction was that this was interesting but non-statistical so I’d have to either post it on the sister blog or wait until the 30 days of statistics [...]
Experimental reasoning in social science
As a statistician, I was trained to think of randomized experimentation as representing the gold standard of knowledge in the social sciences, and, despite having seen occasional arguments to the contrary, I still hold that view, expressed pithily by Box, Hunter, and Hunter (1978) that “To find out what happens when you change something, it [...]
Controversy over the Christakis-Fowler findings on the contagion of obesity
Nicholas Christakis and James Fowler are famous for finding that obesity is contagious. Their claims, which have been received with both respect and skepticism (perhaps we need a new word for this: “respecticism”?) are based on analysis of data from the Framingham heart study, a large longitudinal public-health study that happened to have some social network data (for the odd reason that each participant was asked to provide the name of a friend who could help the researchers locate them if they were to move away during the study period.
The short story is that if your close contact became obese, you were likely to become obese also. The long story is a debate about the reliability of this finding (that is, can it be explained by measurement error and sampling variability) and its causal implications.
This sort of study is in my wheelhouse, as it were, but I have never looked at the Christakis-Fowler work in detail. Thus, my previous and current comments are more along the lines of reporting, along with general statistical thoughts.
We last encountered Christakis-Fowler last April, when Dave Johns reported on some criticisms coming from economists Jason Fletcher and Ethan Cohen-Cole and mathematician Russell Lyons.
Lyons’s paper was recently published under the title, The Spread of Evidence-Poor Medicine via Flawed Social-Network Analysis. Lyons has a pretty aggressive tone–he starts the abstract with the phrase “chronic widespread misuse of statistics” and it gets worse from there–and he’s a bit rougher on Christakis and Fowler than I would be, but this shouldn’t stop us from evaluating his statistical arguments. Here are my thoughts:
An argument that can’t possibly make sense
Tyler Cowen writes: Texas has begun to enforce [a law regarding parallel parking] only recently . . . Up until now, of course, there has been strong net mobility into the state of Texas, so was the previous lack of enforcement so bad? I care not at all about the direction in which people park [...]
Grouponomics, counterfactuals, and opportunity cost
I keep encountering the word “Groupon”–I think it’s some sort of pets.com-style commercial endeavor where people can buy coupons? I don’t really care, and I’ve avoided googling the word out of a general animosity toward our society’s current glorification of get-rich-quick schemes. (As you can tell, I’m still bitter about that whole stock market thing.) [...]
What Do We Learn from Narrow Randomized Studies?
Under the headline, “A Raise Won’t Make You Work Harder,” Ray Fisman writes: To understand why it might be a bad idea to cut wages in recessions, it’s useful to know how workers respond to changes in pay–both positive and negative changes. Discussion on the topic goes back at least as far as Henry Ford’s [...]
Is the internet causing half the rapes in Norway? I wanna see the scatterplot.
Ryan King writes:
This involves causal inference, hierarchical setup, small effect sizes (in absolute terms), and will doubtless be heavily reported in the media.
The article is by Manudeep Bhuller, Tarjei Havnes, Edwin Leuven, and Magne Mogstad and begins as follows:
Does internet use trigger sex crime? We use unique Norwegian data on crime and internet adoption to shed light on this question. A public program with limited funding rolled out broadband access points in 2000-2008, and provides plausibly exogenous variation in internet use. Our instrumental variables and fixed effect estimates show that internet use is associated with a substantial increase in reported incidences of rape and other sex crimes. We present a theoretical framework that highlights three mechanisms for how internet use may affect reported sex crime, namely a reporting effect, a matching effect on potential offenders and victims, and a direct effect on crime propensity. Our results indicate that the direct effect is non-negligible and positive, plausibly as a result of increased consumption of pornography.
How big is the effect?
D. Kahneman serves up a wacky counterfactual
I followed a link from Tyler Cowen to this bit by Daniel Kahneman: Education is an important determinant of income — one of the most important — but it is less important than most people think. If everyone had the same education, the inequality of income would be reduced by less than 10%. When you [...]
Improvement of 5 MPG: how many more auto deaths?
This entry was posted by Phil Price. A colleague is looking at data on car (and SUV and light truck) collisions and casualties. He’s interested in causal relationships. For instance, suppose car manufacturers try to improve gas mileage without decreasing acceleration. The most likely way they will do that is to make cars lighter. But [...]
Bringing Causal Models Into the Mainstream
John Johnson writes at the Statistics Forum.
“Are Wisconsin Public Employees Underpaid?”
Amy Cohen points me to this blog by Jim Manzi, who writes:
Poverty, educational performance – and can be done about it
Andrew has pointed to Jonathan Livengood’s analysis of the correlation between poverty and PISA results, whereby schools with poorer students get poorer test results. I’d have written a comment, but then I couldn’t have inserted a chart. Andrew points out that a causal analysis is needed. This reminds me of an intervention that has been [...]
An IV won’t save your life if the line is tangled
Alex Tabarrok quotes Randall Morck and Bernard Yeung on difficulties with instrumental variables. This reminded me of some related things I’ve written. In the official story the causal question comes first and then the clever researcher comes up with an IV. I suspect that often it’s the other way around: you find a natural experiment [...]
Teaching evaluations, instructor effectiveness, the Journal of Political Economy, and the Holy Roman Empire
Joan Nix writes: Your comments on this paper by Scott Carrell and James West would be most appreciated. I’m afraid the conclusions of this paper are too strong given the data set and other plausible explanations. But given where it is published, this paper is receiving and will continue to receive lots of attention. It [...]
Cars vs. trucks
Anupam Agrawal writes:
I am an Assistant Professor of Operations Management at the University of Illinois. . . . My main work is in supply chain area, and empirical in nature. . . . I am working with a firm that has two separate divisions – one making cars, and the other makes trucks. Four years back, the firm made an interesting organizational change. They created a separate group of ~25 engineers, in their car division (from within their quality and production engineers). This group was focused on improving supplier quality and reported to car plant head . The truck division did not (and still does not) have such an independent “supplier improvement group”. Other than this unit in car, the organizational arrangements in the two divisions mimic each other. There are many common suppliers to the car and truck division.
Data on quality of components coming from suppliers has been collected (for the last four years). The organizational change happened in January 2007.
My focus is to see whether organizational change (and a different organizational structure) drives improvements.
Regression discontinuity designs: looking for the keys under the lamppost?
Jas sends along this paper (with Devin Caughey), entitled Regression-Discontinuity Designs and Popular Elections: Implications of Pro-Incumbent Bias in Close U.S. House Races, and writes:
The paper shows that regression discontinuity does not work for US House elections. Close House elections are anything but random. It isn’t election recounts or something like that (we collect recount data to show that it isn’t). We have collected much new data to try to hunt down what is going on (e.g., campaign finance data, CQ pre-election forecasts, correct many errors in the Lee dataset). The substantive implications are interesting. We also have a section that compares in details Gelman and King versus the Lee estimand and estimator.
I had a few comments:
Quality control problems at the New York Times
I guess there’s a reason they put this stuff in the Opinion section and not in the Science section, huh? P.S. More here.
Estimating the effect of A on B, and also the effect of B on A
Lei Liu writes:
Matching for preprocessing data for causal inference
Chris Blattman writes: Matching is not an identification strategy a solution to your endogeneity problem; it is a weighting scheme. Saying matching will reduce endogeneity bias is like saying that the best way to get thin is to weigh yourself in kilos. The statement makes no sense. It confuses technique with substance. . . . [...]
Is instrumental variables analysis particularly susceptible to Type M errors?
Hendrik Juerges writes: I am an applied econometrician. The reason I am writing is that I am pondering a question for some time now and I am curious whether you have any views on it. One problem the practitioner of instrumental variables estimation faces is large standard errors even with very large samples. Part of [...]
Story time
This one belongs in the statistical lexicon. Kaiser Fung nails it: In reading [news] articles, we must look out for the moment(s) when the reporters announce story time. Much of the article is great propaganda for the statistics lobby, describing an attempt to use observational data to address a practical question, sort of a Freakonomics-style [...]
Randomized experiments, non-randomized experiments, and observational studies
In the spirit of Dehejia and Wahba: Three Conditions under Which Experiments and Observational Studies Produce Comparable Causal Estimates: New Findings from Within-Study Comparisons, by Cook, Shadish, and Wong. Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments, by Shadish, Clark, and Steiner. I just talk about causal inference. These [...]