XL sent me this paper, “A Fruitful Resolution to Simpson’s Paradox via Multi-Resolution Inference.”
I told Keli and Xiao-Li that I wasn’t sure I fully understood the paper—as usual, XL is subtle and sophisticated, also I only get about half of his jokes—but I sent along these thoughts:
1. I do not think counterfactuals or potential outcomes are necessary for Simpson’s paradox. I say this because one can set up Simpson’s paradox with variables that cannot be manipulated, or for which manipulations are not directly of interest.
2. Simpson’s paradox is part of a more general issue that regression coefs change if you add more predictors, the flipping of sign is not really necessary.
Here’s an example that I use in my teaching that illustrates both points:
I can run a regression predicting income from sex and height. I find that the coef of sex is $10,000 (i.e., comparing a man and woman of the same height, on average the man will make $10,000 more) and the coefficient of height is $500 (i.e., comparing two men or two women of different heights, on average the taller person will make $500 more per inch of height).
How can I interpret these coefs? I feel that the coef of height is easy to interpret (it’s easy to imagine comparing two people of the same sex with different heights), indeed it would seem somehow “wrong” to regress on height _without_ controlling for sex, as much of the raw difference between short and tall people can be “explained” by being differences between men and women. But the coef of sex in the above model seems very difficult to interpret: why compare a man and a woman who are both 66 inches tall, for example? That would be a comparison of a short man with a tall woman. All this reasoning seems vaguely causal but I don’t think it makes sense to think about it using potential outcomes.
There are variables which cannot be manipulated by a human, but it is still possible to imagine hypothetical manipulations. For example, we discuss how the color of a certain species of a plant is not really manipulable (it comes with the species). But one can imagine some sort of “proto”-plant which is that plant before it had color attached. We can now talk about manipulation of color for the proto-plant. The key however is that our unit of analysis has changed: when we think about manipulation of color, our fundamental unit is no longer plant but rather proto-plant. So we can always imagine hypothetical manipulations, but we need to keep careful accounting of what our unit of analysis is.
Now to the second part of the question: why should we be interested in these hypothetical manipulations in the first place if we cannot perform the actual manipulation? As you say, Simpson’s Paradox is really encompassed by the more general problem of which predictors to include in our regression. Imagine the ideal scenario where we can gather data from as many individuals as we want and for each individual we have an incredibly rich set of predictors. Now suppose we want to study the effect of height on income. What predictors should we include in our regression? Our point is that to answer this question, you need to think about manipulation of height, even if it is only a hypothetical manipulation.
In particular, when we think about the hypothetical manipulation of height, our unit of analysis is no longer an individual from the dataset (since these individuals come with height already). Rather the unit of analysis is some “proto”-individual. These “proto”-individuals have attributes but height is not one of these attributes—hence why we can think about manipulation of height. We should include in the regression all predictors which are attributes of this “proto”-individual, but leave out predictors which are not attributes of this “proto”-individual.
When you say “indeed it would seem somehow ‘wrong’ to regress on height _without_ controlling for sex”, what you are implicitly doing in your mind is conceiving of this “proto”-individual and realizing that sex is in fact an attribute of this “proto”-individual. Similarly, when you say, “why compare a man and a woman who are both 66 inches tall, for example?” what you have done is to imagine a hypothetical manipulation of gender. In your thought experiment, you have made the judgment call that the type of proto-individual which allows a hypothetical manipulation of gender does not possess the attribute of height (if we are thinking about God making humans on an assembly line, height is added only after gender) [No, I'm not thinking about God making humans on an assembly line -- AG]. Hence we should not include height in the regression if our goal is to learn about the effect of gender.
To me, the whole proto-individual idea just adds more complexity and leaves me even more confused! And I also think I can interpret those regressions without having to think about manipulation of height or of sex—to me, these are between-person comparisons, not requiring within-person manipulations. But I’ll put this all out there for the rest of you to chew on.