Some thoughts on survey weighting

From a comment I made in an email exchange:

My work on survey adjustments has very much been inspired by the ideas of Rod Little. Much of my efforts have gone toward the goal of integrating hierarchical modeling (which is so helpful for small-area estimation) with post stratification (which adjusts for known differences between sample and population). In the surveys I’ve dealt with, nonresponse/nonavailability can be a big issue, and I’ve always tried to emphasize that (a) the probability of a person being included in the sample is just about never known, and (b) even if this probability were known, I’d rather know the empirical n/N than the probability p (which is only valid in expectation). Regarding nonparametric modeling: I haven’t done much of that (although I hope to at some point) but Rod and his students have.

As I wrote in the first sentence of the above-linked paper, I do think the current theory and practice of survey weighting is a mess, in that much depends on somewhat arbitrary decisions about which variables to include, which margins to weight on, and how to trim extreme weights. And that’s just for estimating population totals. Once we move to regressions, weighting becomes even messier. This is not to say that weighting should not be done—I construct survey weights myself sometimes—but I think it’s important to recognize that the theory has been holes, it’s not a simple matter of clean unbiased estimates as is sometimes presented in introductory presentations and even to users.

P.S. In response to a comment, I elaborated:

It’s hard for me to see how anyone who has actually constructed survey weights can disagree with the statement that survey weighting is a mess. But I suppose one could also say that regression modeling is a mess. In both cases there is so much awkwardness surrounding the steps of selecting or rejecting variables for the adjustment, and there are so many potential interactions while at the same time big practical limitations on how much can be included using the least-squares-like estimation and adjustment procedures that are standard. To me, survey weighting seems even messier than regression modeling in that survey weighting involves two additional complications: (1) the numbers that we want to adjust to (census totals or sampling probabilities) are commonly not themselves known as precisely as we would like, and (2) there’s that whole business of trimming weights.

6 thoughts on “Some thoughts on survey weighting

  1. The late David Binder told me that was asked to review that paper. He decided not to because he just couldn’t get past that first sentence. I gathered from his tone and body language that strongly disagreed.

    • Corey:

      It’s hard for me to see how anyone who has actually constructed survey weights can disagree with the statement that survey weighting is a mess. But I suppose one could also say that regression modeling is a mess. In both cases there is so much awkwardness surrounding the steps of selecting or rejecting variables for the adjustment, and there are so many potential interactions while at the same time big practical limitations on how much can be included using the least-squares-like estimation and adjustment procedures that are standard. To me, survey weighting seems even messier than regression modeling in that survey weighting involves two additional complications: (1) the numbers that we want to adjust to (census totals or sampling probabilities) are commonly not themselves known as precisely as we would like, and (2) there’s that whole business of trimming weights.

      I’d be interested in knowing how Binder (or others) would disagree with the above.

        • Corey:

          I had a few conversations with David (who sadly passed recently).

          He seldom said more while indicating disagreement and seemingly being exercised on the topic.

          He _seemed_ to have strong views and I suspect understood design based inference better than perhaps anyone else.

          I don’t think he felt weights and weighting would be definitly sorted out in most applications – but he is _rudely_ unavailable to ask.

  2. I certainly don’t disagree with this post; instead I’ll add one comment: often the results of an analysis do not depend strongly on the weights, in which case you can probably ignore this issue. In the case that the weights matter a lot, (1) you should put some real effort into seeing what is causing the difference and try to understand it substantively rather than just statistically, and (2) recognize that you are not really going to get the weights right for the messy reasons Andrew alludes to, so your statistical uncertainties are going to be wider — maybe a lot wider — than a canned analysis spits out.

  3. But there is an upside to working on weighting problems, I’ve found. If this is your job, nobody wants it. The CEO can come down personally to your office and curse you out using terms that will kill your plants, but nobody will step up and say “I want that job”.

Comments are closed.