Debate on using margin of error with non-probability panels

Tomorrow (Thurs 22 Jan) at 2pm, I’m participating (along with Jane Tang, John Bremer, Nancy Brigham, and Steve Mossup) on an online discussion, moderated by Annie Pettit, on the above topic.

Here’s the description:

Most marketing researchers know that using Margin of Error with convenience samples, non-probability samples, and online research panels is inappropriate. However, some researchers continue to report MOE as there does not seem to be a simple or any alternative.

Join Ipsos and a panel of experts for a webinar discussion about:

Why is it appropriate or inappropriate to use MOE with online research panels?

Is it appropriate to use MOE with other types of research, e.g., telephone surveys / RDD

Are there any appropriate alternatives that give similar guidance?

If there are no appropriate alternatives, what should researchers do to guide people interpreting their data?

How can researchers/pollsters who do not use MOE compete with pollsters who do use MOE, particularly when research users demand it?

How are research users supposed to know good from bad without seeing MOE or alternatives?

I can’t tell you how much I hate that first sentence in the above blurb. “Most marketing researchers know” should be replaced by “Many marketing researchers believe.”

I don’t really know what I’ll have to say, beyond yapping out “91% nonresponse! 91% nonresponse!” like a demented version of Long John Silver’s parrot. Any of you who want all my content without hearing the discussion can read this post or this article.

Anyway, the panel will be 30-45 minutes long, and it seems that you can sign up here. Too bad they didn’t get Michael Link, president of AAPOR, to participate; then I could’ve asked him why he didn’t respond to my request for clarification.

P.S. Due to technical difficulties this event never happened. It got rescheduled to a time next week that I can’t make, but you can go hear the others, I suppose. I’ll post something tomorrow on what we did say during our brief panel discussion. And here it is.

10 thoughts on “Debate on using margin of error with non-probability panels

  1. “… marketing researchers know that using Margin of Error with convenience samples, non-probability samples, and online research panels is inappropriate”

    Well, I’m a marketing researcher and I don’t know that. Because it’s wrong.

    The margin of error around a convenience sample is the margin of error around a similarly drawn convenience population. If you are interviewing people in an upscale mall on a Thursday afternoon, you are drawing a margin of error around an estimate for a population of people who agreed to be interviewed in this upscale mall on a Thursday afternoon.

    Yes, there’s an issue here in terms of whether that’s of any use to you, but it’s still a correct margin of error.

    It’s just that you probably want the margin of error for the inference about a different population.

    Furthermore, even if you had a perfect probability sample, you have the issue of whether the respondent’s answers can be trusted. The old rule of thumb used to be if a respondent said they would “definitely buy” a product, they had a 75% chance of buying. “Probably buy” meant a 25% chance. This calibration meant that if you took the responses literally (e.g. definitely as 100%, probably as 51%) you would have a margin of error which would not be anywhere near what the client really wanted to know about the population — i.e. what’s the forecasted sales.

    It’s these needs for adjustment (in the example of purchase intent) and inference (as we go from fallible sample to population) that mean we have to take any MOE with a grain of salt if we are applying the MOE to what the client really wants to know.

    How do we solve this? By building enough of a structure to provide methods of adjustment, a structure that allows us to feel confident because this structure has allowed us to predict past weather accurately, past elections accurately, predict past purchases accurately, and so on. It’s that structure that allows us to approach a MOE that gets to the MOE that the client wants.

    Are the MOEs around convenience samples useful? Not very, but they have their uses. For example, in evaluating whether there have been changes over time in two convenience samples collected similarly, or if there is an experimental design embedded in the survey.

  2. Full disclosure: the 75/25 rule goes back to old ARF (Advertising Research Foundation) Arrowhead work. I also worked for many years (and still consult for) a competitor of Ipsos.

  3. I haven’t been a member in AAPOR for several years and wasn’t very active when I was a member, but I think Link’s statement is a fairly typical sentiment there. If I read into conversations I’ve had with folks active in AAPOR, a big misconception seems to be equating known probability sampling approach with having a known probability sample. If you collect the necessary data, you can calculate the probability of selection under ideal conditions (i.e. no non-response). However, when you have large non-response you in all likelihood do not know the selection mechanism that determined responder vs. non-responder. I argue that you no longer have a true “known probability sample” since you would need to know both the pr(select) due to design as well as self-selection. Of course, we make practical adjustments to make the sample composition more like the Census (or some other markers of your population), but that’s not a direct model of the selection mechanism and only works to the extent the variables you post-stratify on are correlated with the self-selection mechanism.

    To extend this to standard error calculation, I don’t see that big of a difference between opt-in vs. random-digit dialing (or area probability, etc.). In both designs, I would argue, do you end up with unknown probability samples except in the rare case where you have virtually no non-response. You just have a bit more information using a known probability based sampling approach, such as the RDD. In either design is it reasonable to make practical adjustments through post-stratification. Perhaps the models look different, but you can end up with reasonable estimates either way if you have reasonable models. It goes back to knowing your sample, subject, and population.

  4. I can remember going to an ARF conference in NYC in the 90s when 80%-90% of market research was done using mall intercept-based nonprobability samples and hearing Lester Frankel thunder from the podium about how stupid market researchers were to project to the population based on this convenience sample data. He was right then and in today’s internet intercept-based research, he’s still right. Things haven’t changed that much.

    Today’s workaround is to reweight any bias in the demographics of the convenience sample to, e.g., Census population marginals…and call that projectable. These are heuristics: kluge, swag, rules of thumb…nothing more nor less.

    But then, as Joseph Kadane has noted, p<=.05 as a significance level is a similar rule of thumb with no theoretical motivation or foundation, a convention inherited from Fisher and agreed upon by statisticians.

    Today's kluge is tomorrow's best practice, eh?

    • > Today’s kluge is tomorrow’s best practice, eh?
      In the land of the blind the one eyed man is king! (Kipling, I think)

      In the land of the bind the one eyed man is king! (Me – due to a typo)

    • Kyle-That is absolutely true. But Kadane’s comment was in a JASA paper he wrote on forensic statistics from the early 90s…so, let’s give him some precedence…

Leave a Reply

Your email address will not be published. Required fields are marked *