Don’t believe the bounce

image001

Alan Abramowitz sent us the above graph, which shows the results from a series of recent national polls, for each plotting Hillary Clinton’s margin in support (that is, Clinton minus Trump in the vote-intention question) vs. the Democratic Party’s advantage in party identification (that is, percentage Democrat minus percentage Republican).

This is about as clear a pattern as you’ll ever see in social science: Swings in the polls are driven by swings in differential nonresponse. After the Republican convention, Trump supporters were stoked, and they were more likely to respond to surveys. After the Democratic convention, the reverse: Democrats are more likely to respond, driving Clinton up in the polls.

David Rothschild and I have the full story up at Slate:

Tou sort of know there is a convention bounce that you should sort of ignore, but why? What’s actually in a polling bump? The recent Republican National Convention featured conflict and controversy and one very dark acceptance speech—enlivened by some D-list celebrities (welcome back Chachi!)—but it was still enough to give nominee Donald Trump a big, if temporary, boost in many polls. This swing, which occurs predictably in election after election, is typically attributed to the persuasive power of the convention, with displays of party unity persuading partisans to vote for their candidate and cross-party appeals coaxing over independents and voters of the other party.

Recent research, however, suggests that swings in the polls can often be attributed not to changes in voter intention but in changing patterns of survey nonresponse: What seems like a big change in public opinion turns out to be little more than changes in the inclinations of Democrats and Republicans to respond to polls. We learned this from a study we performed [with Sharad Goel and Wei Wang] during the 2012 election campaign using surveys conducted on the Microsoft Xbox. . . .

Our Xbox study showed that very few respondents were changing their vote preferences—less than 2 percent during the final month of the campaign—and that most, fully two-thirds, of the apparent swings in the polls (for example, a big surge for Mitt Romney after the first debate) were explainable by swings in the percentages of Democrats and Republicans responding to the poll. This nonresponse is very loosely correlated with likeliness to vote but mainly reflects passing inclinations to participate in polling. . . . large and systematic changes in nonresponse had the effect of amplifying small changes in actual voter intention. . . .

[See this paper, also with Doug Rivers, with more, including supporting information from other polls.]

We can apply these insights to the 2016 convention bounces. For example, Reuters/Ipsos showed a swing from a 15-point Clinton lead on July 14 to a 2-point Trump lead on July 27. Who was responding in these polls? The pre-convention survey saw 53 percent Democrats, 38 percent Republican, and the rest independent or supporters of other parties. The post-convention respondents looked much different, at 46 percent Democrat, 43 percent Republican. The 17-point swing in the horse-race gap came with a 12-point swing in party identification. Party identification is very stable, and there is no reason to expect any real swings during that period; thus, it seems that about two-thirds of the Clinton-Trump swing in the polls comes from changes in response rates. . . .

Read the whole thing.

The political junkies among you have probably been seeing all sorts of graphs online showing polls and forecasts jumping up and down. These calculations typically don’t adjust for party identification (an idea we wrote about back in 2001, but without realizing the political implications that come from systematic, rather than random, variation in nonresponse) and thus can vastly overestimate swings in preferences.

13 thoughts on “Don’t believe the bounce

  1. For another take on this phenomenon, consider my remarks on http://statmodeling.stat.columbia.edu/2016/02/01/peer-review-make-no-damn-sense/#comments (referencing http://statmodeling.stat.columbia.edu/2014/09/30/didnt-say/.
    Essentially, I consider polling estimates of party identification to be as variable as estimates of voting choice, due to simple sampling error. Here is a brief synopsis of my criticism:

    If you recall the 2000 election, the polls varied dramatically over the course of the campaign (see
    http://en.wikipedia.org/wiki/Historical_polling_for_U.S._Presidential_elections#United_States_presidential_election.2C_2000). Yet there was no indication of large non-response problems by party from any of the polling organizations (maybe there was and I didn’t see it/it wasn’t reported, but my impression is there wasn’t and I didn’t see any of this in the polling I was doing). This is why I think your findings are probably not true (slump versus changing responses). Also, when you state that a 5% difference is “major”, you are ignoring that the Pew estimates are random and that the 55 to 47 percent difference in two successive samples may very well be within expected sampling error bounds, depending upon the Pew n (which I
    don’t know). Isn’t ignoring randomness in estimation a type M error?

    I’m not saying I’m right but I don’t see evidence which rules out a) sampling error in polling party id and b) small perturbations in party id (I know party id is supposed to be fixed–hello American Voter–but pollsters often don’t treat it that way, treating it as endogenous to vote choice).

    • Numeric:

      1. There’s definitely sampling error in polling party ID; we never claimed otherwise.

      2. In our Xbox survey there were no fluctuations in the party ID response by definition, as we asked the party ID question at the beginning of the survey period, so it was super-clear that there were very strong patterns in differential nonresponse, with relatively more response from Democrats when Obama was doing well and relatively more response from Republicans when Romney was doing well. Also in our 1993 paper we looked at the party ID question and we did not see evidence of big swings in party ID. So when I see a graph such as the one above, I have strong doubts that this is being driven by people changing which party they identify with.

      • In our Xbox survey there were no fluctuations in the party ID response by definition, as we asked the party ID question at the beginning of the survey period.

        You asked Party ID once and did not ask again, correct (this is how I interpret your “by definition” comment)? That means Party ID may have changed and you would not be able to measure that (and with the heightened sensitivity of panel respondents one would expect much less change if you had tried to measure it). I examine http://www.gallup.com/poll/15370/party-affiliation.aspx and I see that PID changes up to 6 points in a month (see May 18-22, 2016 Republicans plus leaners (47%) as opposed to April 6-10, 2016 (41%)). I don’t know how you define no evidence of big swings but a 6% change is more than enough to explain the three clusters in Abramowitz’s graph–July 26th and before, July 27, 28 and 29, and July 30 and later. While that is over a month and this was 9 days the conventions are very salient so it seems very plausible we might see a similar shift (Gallop apparently only asks it once a month so that data isn’t available to address this question).

        We also still have the case of the 2000 election (described in detail in my linked comments) where the polling averages varied dramatically over time and it is almost certainly _not_ non-response. These two observations plus the comment of pollsters such as Dick Morris that they treat PID as endogenous cause me to think the non-response explanation is not the correct one.

        • Numeric:

          The Xbox survey does not directly supply evidence on party ID changing or not changing; it provides evidence that differential nonresponse is large and real, and that changes in differential nonresponse can cause big swings in the horse-race numbers even while only a very small percentage of people are actually changing their vote preferences.

          I’m willing to go with our evidence on this rather than trust the statements of Dick Morris or Mark Penn or Pat Cadell or whatever.

        • I’ve already explained how actually to test your hypothesis of non-response versus PID change (http://statmodeling.stat.columbia.edu/2014/09/30/didnt-say/) by sampling for representative voters. If it is changes in PID/vote choice, then the polling in that method will reflect the “bounce” phenomenon. If it is non-responsiveness, then the bounce phenomenon will not occur.

          Here is the exact procedure:

          As regard to the applicability of your paper to pollsters, in states where you can obtain registered voter list, a very simple way to ensure non-biasness of response by partisan/demographic categories is to sort the registered voter list by various criteria (partisan id, sex, age, ethnicity (through surname matching)) and then create clusters of, say, 50 names for 400, 800, or however many n you want to contact. This gives a rough equivalence to the actual composition of the district/state and you start with number one in the cluster and call until you get a respondent. There might be increased non-response in some of these clusters after a campaign event (I’ve never looked) but even if you have to call 4 names in a democratic cluster (say) as opposed to 3 in a republican cluster (presume this is after a republican convention), it doesn’t matter (unless you want to claim there is a vote-choice bias in the democrats that do respond as opposed to those who have “slumped”–I’ve never seen this either). Anyway, this is why non-response isn’t typically a problem in states where you can get registered voters with party identification (obviously, you have to pay to get the phone numbers matched since they typically won’t be on the registered voter file).

  2. It seems plausible to me that survey non-response due to lack of enthusiasm might predict non-voting due to lack of enthusiasm, in which case swings in the polls would not reflect changes in partisan make-up of the electorate but would reflect changes in likely election outcomes.

    • Corey:

      We discuss this in our paper. Short answer: voter turnout in presidential elections is about 60%. Response rates in polls is less than 10%. There’s a lot more room for differential nonresponse in polls than in the general election.

  3. “This nonresponse…mainly reflects passing inclinations to participate in polling.”

    I agree with Martha. Very elegant.

    I like the idea of using this predictable selection bias to generate variation in polling quality, and then using that variation to investigate whether/how people respond differently to true and false information. Follow-up possibilities:

    1 – The returns to luck in the market for opinion polling: The local political climate alters the probability of response differentially by party ID, generating variation in poll accuracy over time and space… are there returns (in terms of exposure, prestige, links, likes, retweets, times heard discussed on subway commute) to accuracy? Or to some or all types of exaggeration?

    2 – The effect of noise from polling on campaigning and public policy: do noise from polls and signal from polls have the same influence on the issues candidates address or the policy positions they take? Do candidates and campaigns and public officials make policy decisions based on false information about voter preferences?

    3 – The effect of perceived peer attitudes on individuals’ actual voting intentions – with the X-box data, you can watch people’s preferences as ridiculously wrong polls come in and see if they sway people one way (bandwagon hoppers) or the other (contrarians).

  4. I wonder how this might affect some of the experimental findings we’ve seen around campaign advertising. The gist of that work seems to be that advertising has relatively small effects (1-2 points) that disappear within a few days. However, it could very well be the case that a portion of that is just selective non-response.

  5. I would need a lot of convincing before I believe that party ID is actually constant and not heavily tied to how much someone thinks they’re likely to vote for the Democrat or Republican candidate right now.

Leave a Reply to Andrew Cancel reply

Your email address will not be published. Required fields are marked *