Mark Blumenthal writes:
What do you think about the “random rejection” method used by PPP that was attacked at some length today by a Republican pollster. Our just published post on the debate includes all the details as I know them. The Storify of Martino’s tweets has some additional data tables linked to toward the end.
Also, more specifically, setting aside Martino’s suggestion of manipulation (which is also quite possible with post-stratification weights), would the PPP method introduce more potential random error than weighting?
From Blumenthal’s blog:
B.J. Martino, a senior vice president at the Republican polling firm The Tarrance Group, went on an 30-minute Twitter rant on Tuesday questioning the unorthodox method used by PPP [Public Policy Polling] to select samples and weight data: “Looking at @ppppolls new VA SW. Wondering how many interviews they discarded to get down to 601 completes? Because @ppppolls discards a LOT of interviews. Of 64,811 conducted for @DailyKos /SEIU in 2012, they discarded almost 23K. Sure, a handful of the @ppppolls discards were not valid interview responses. Most appear valid completes. @ppppolls says discarding interviews is a kind of retroactive quota on race, gender and age. Why just not weight the data? . . .
PPP’s explanation of how they weight data – PPP’s explanation of their method appears on their “About Us” page: “Our first step in weighting is to survey more than enough people. This allows us to then be able to randomly reject individual surveys from demographics that are overrepresented. For example, in our polling more women answer relative to men, and not enough African-Americans answer our surveys. Our random selection eliminates any potential bias from the rejections, plus it functions like a quota, only after the fact. PPP also employs a mathematical weighting scheme that assigns a weight based on each demographic.” . . .
Via email, Martino clarifies: “The random process they use to discard older white female interviews in this case, changes the reported composition of and the opinions of the older white females who remain. The discard process can be (can be, not saying is) manipulated to produce desired results. Even random discards within a selected sub-group can be the result of choices the pollster made. Ultimately, why discard at all when you are already weighting after the fact?”
Response from PPP – In response to an email query, PPP’s Tom Jensen defended his company but not the specific methodology challenged by Martino: “I’m sure there are as many methods for weighting polls as there are polling companies. We’ve been doing things the way we do them for over a decade and it’s served us well. . . .
I admit to being a bit baffled by all of this. If this organization is actually going to the trouble of doing full survey interviews on these people, then they definitely shouldn’t be throwing away the responses. Maybe they’re doing some sort of screening, where they’re only asking a few questions and using these in order to decide whether to go on? That could make sense. The whole thing seems so odd to me that I wonder if I’m missing something here.