Is Rigor Contagious?
Much of the theory and practice of statistics and econometrics is characterized by a toxic mixture of rigor and sloppiness. Methods are justified based on seemingly pure principles that can’t survive reality. Examples of these principles include random sampling, unbiased estimation, hypothesis testing, Bayesian inference, and causal identification. Examples of uncomfortable reality include nonresponse, varying effects, researcher degrees of freedom, actual prior information, and the desire for external validity. We discuss a series of scenarios where researchers naively think that rigor in one part of their design and analysis will assure rigor on their larger conclusions, and then we discuss possible hierarchical Bayesian solutions in which the load of rigor is more evenly balanced across the chain of scientific reasoning.
The talk (for the Sustainable Development seminar) will be Mon 27 Feb, 4:15-5:45, in room 801 International Affairs Building at Columbia.
Any plans to publish slides or a white paper based on this talk?
Sentinel:
I’ve been phasing out slides—that’s the topic of a future post—but I do plan to write a paper on this. Haven’t done it yet, though.
May be this quote from Tolstoy fits: “Happy families are all alike; every unhappy family is unhappy in its own way.” (from Anna Karenina)
By focusing on that list of pure principles, we are focusing on happy families — what we should practice if things were perfect. There is one happy family of random sampling methods, for example, and a small set of estimators and variance estimates to go with those sampling methods.
But every set of real data problems is likely to be different. In some areas, we have a decent characterization of the ways they can be different and what to do about them (I’d put missing data in this class). But each data set has its own potential pitfalls and its own potential leverage points.
An example of unusual pitfalls: How many of you have to check that the data wasn’t sent to you in octal, rather than decimal? When there is a “-1” in the unit sales field, does that mean a return, does that mean a coupon discount, etc. and how does that vary by store? Similarly, nonresponse biases and question wording and question context biases are known to exist, but both understanding the problem and figuring out how to adjust can be difficult.
That’s perhaps why we cling to the fictions of a ‘happy family’ research.
“