Using randomized incentives as an instrument for survey nonresponse?

I received the following question:

Is there a classic paper on instrumenting for survey non-response? some colleagues in public health are going to carry out a survey and I wonder about suggesting that they build in a randomization of response-encouragement (e.g. offering additional $ to a subset of those who don’t respond initially). Can you recommend a basic treatment of this, and why it might or might not make sense compared to IPW using covariates (without an instrument)?

My reply: Here’s the best analysis I know of on the effects of incentives for survey response. There have been several survey-experiments on the subject. The short answer is that the effect on nonresponse is small and the outcome is highly variable, hence you can’t very well use it as an instrument in any particular survey.

My recommended approach to dealing with nonresponse is to use multilevel regression and poststratification; an example is here.

Inverse-probability weighting doesn’t really work because you don’t know the probability of nonresponse; all you really can do, usually, is poststratify. I discuss some of the issues here and here.

11 thoughts on “Using randomized incentives as an instrument for survey nonresponse?

  1. “My recommended approach to dealing with nonresponse is to use multilevel regression and poststratification”

    `

    Is that approach valid for all levels of nonresponse ?

    If not, is there some formal/informal nonresponse threshold at which that approach can be confidently employed ?

    Is there any nonresponse rate threshold (below 100%) that would manifestly, in itself, invalidate a normal survey research product ?

    • Whether or not nonresponse is a problem is not a function of the level of nonresponse, but rather the extent to which nonresponse is correlated with your outcome variable of interest and whether or not you have adequate auxiliary variables on which to poststratify. If nonresponse is uncorrelated with your variable of interest, then the level doesn’t matter (for bias). If it is correlated, but you have good information for postratification, then the level also doesn’t matter (although it’s almost impossible to ever be sure about this). The most you can say is that low nonresponse reduces the risk of bias but it doesn’t eliminate it, and that it really depends on what you’re trying to measure and how precise you need your measurement to be.

  2. A reasonably common practice in field surveys in development economics is the following: First run the survey and see how much nonresponse there is. If the nonresponse is large or seems nonrandom (based on observables), then randomly choose some of the nonrespondents and search for them in an intensive way. The intensive search is then an instrument. I believe Kremer and Miguel’s worms papers does this.

  3. I really don’t think it’s a good idea to pay non-responders to respond because 1) if the responders find out they’ll be really annoyed that their good will was so little valued and 2) once the non-responders start blabbing about their good fortune, it will turn future responders into non-responders in the hopes of a similar reward (which will drive up the time costs of interviews and increase the need for more reward money).

      • I think the suggestion here is a randomized incentive, with the idea that (i) the incentive will encourage some response among those who would not respond otherwise, (ii) as the incentive was randomized, we can make inferences about the survey responses of the ‘incentive-sensitive’ group (i.e. participate in survey iff receive incentive), and (iii) we can extrapolate from this group to everyone who did not respond. It seems this last step is the hairy one, unless the incentive brings in almost all the non-responders.

  4. I’m stats stupid (honest, I just surfed in here for a different reason), so naïve question. Is there some way to supplement surveys (with low response) with some more in depth measures (for example door to doors versus phone) or case studies versus market research. Obviously, since it is a higher effort measure, you would want to do less data points. And then I don’t know if it tells you anything. Can you use it to somehow test your larger survey where the weak points are (e.g. testing out the whole “cell phone” crit) and do so in a cheap, info-adding manner?

  5. From an entirely unrelated field (which frequently deals with high non-response and attrition) comes this article:

    http://www.jmir.org/2011/1/e26/

    ” In study 1 (n = 1226), there was no significant difference in response rates between those participants offered an incentive (175/615, 29%) and those with no offer (162/611, 27%) (difference = 2%, 95% confidence interval [CI] –3% to 7%). There was no significant difference in response rates among the three different incentives offered. In study 2 (n = 2591), response rates were 9% higher in the group offered an incentive (476/1296, 37%) than in the group not offered an incentive (364/1295, 28%) (difference = 9%, 95% CI 5% to 12%, P < .001). The incremental cost per extra successful follow-up in the incentive arm was £110 in study 1 and £52 in study 2."

    Also of interest is this systematic review:

    http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=111107&tool=pmcentrez&rendertype=abstract

  6. It sounds like the question is about Heckman-type selection models. If so a ‘classic paper’ is probably Heckman, J. (1979). “Sample selection bias as a specification error”. Econometrica 47(1):153–61.

    In my field this approach has been used to correct for non-response in HIV prevalence surveys. The suspicion is that those survey respondents who don’t provide blood samples to the survey team are more likely to be HIV positive, conditional on other covariates. Recently some researchers have used interviewer ID as the kind of instrument the question asks for. The results generally produce higher HIV prevalence estimates, but also much larger standard errors (appropriately so, I think). For example: http://www.ncbi.nlm.nih.gov/pubmed/21150352.

    These selection models are definitely not the accepted approach yet. It seems to me that if the results of the selection model analysis are roughly the same as the conventional analysis, then one can have more faith in the original result. If the results diverge, then we should be suspicious of any result until we are able to increase participation rates.

  7. With colleagues, we recently realized that you don’t actually need to randomize some incentive to answer surveys (or some survey effort) in order to generate an instrument to correct for non-response. It is enough to make as much survey effort as possible on the FULL sample, and record the number of attemps after which every individual has been reached (if he has). Information on how much effort it took to survey each person is sufficient for identification.

    However, the parameter of interest (say a treatment impact) is generally only bounded: point identification only obtains in very specific cases. Well, in fact this holds also if your random “survey instrument” is discrete, so this is a general feature of selectivity models, sometimes overseen.

    The paper, joint with Luc Behaghel, Bruno Crépon and Thomas Le Barbanchon is at: http://www.iza.org/en/webcontent/publications/papers/viewAbstract?dp_id=6751

Comments are closed.