Nadia Hassan writes:
Justin Wolfers and Nate Silver got into a colorful fight on twitter. Nate has 2 forecasts. Nate is doing a polls-only forecast in addition to a “traditional” one that discounts poll leads and builds in fundamentals. Wolfers noted that the 538 polls-only model had Clinton at a higher chance of winning on August 9th than today, even though arguably a lot of the uncertainty has gone. He opined that the 538 model was broken. Nate sees Wolfers critiques as lazy, esp. with the potential for overfitting in a low-n environment.
On August 9, 538 rated Clinton an 87.6% to win. Today, it's only 86.1%. Is it really plausible her odds haven't improved since then? pic.twitter.com/hazgnIxF0H
— Justin Wolfers (@JustinWolfers) October 25, 2016
: "Discount early leads" (polls-plus) is probably a good assumption. Polls-only *deliberately does not make that assumption*.
— Nate Silver (@NateSilver538) October 25, 2016
I was wondering what you thought about this round of “the nerd fight”. One potential issue with polling error, at the moment, is the nonresponse bias. YouGov is showing a slightly narrower Clinton lead than other national polls are and Ben Lauderdale reported during a big story that Republicans were responding less. I am not sure if that is still the case. But, it is a potential issue.
My reply:
I really really don’t like twitter as a means of communication in this way. It encourages snappiness and discourages engagement with details. Wolfers and Silver are thoughtful and knowledgeable, but I don’t think twitter brings out the best in anyone.
On your final question, yes, given the bad news for Republicans in recent weeks, I’d guess that Republicans are less likely to respond to polls right now, hence I think that if the election were held today, Clinton would not do so well as it appears from the polls.
Full disclosure: Some of my work is supported by YouGov.
I re-looked at The Mythical Swing Voter, (http://www.stat.columbia.edu/~gelman/research/unpublished/swing_voters.pdf ) and I’m concluding something slightly different than you – I’d love to hear your thoughts.
Perhaps I am misunderstanding the implications of nonresponse bias as it relates to elections, but I would assume variable nonresponse should correlate with variable enthusiasm among supporters; if the candidate is polling badly right before the polls open, the phenomena that drive response rates could very easily drive voting behavior as well. (This makes sense given the very high predictive power of polls immediately before an election.)
(My mental model has both voting behavior and polling responses as a function of candidate support, (with different variance,) where each has “nonresponse bias.” This seems to be instead of having polling response be a function of the later-observed voting outcome, which seems to be the assumption of your model. But I may be misunderstanding.)
PS. RE: communicating on Twitter – Twitter has excellent uses, especially for academic communication, but having nuanced public discussion about misunderstandings isn’t one of them.
Additionally, in terms of twitter etiquette, Wolfers didn’t send a tweet directly to Gelman, he made a public comment, cc’ing him. To me, that seems a little bit rude (perhaps unintentionally) on Wolfer’s part. Gelmans response was a reply, so that most of his followers aren’t seeing him explain why he thinks Wolfers was wrong; it’s still public, but it’s not nearly as aggressively so.
David:
1. Voter turnout rate in the general election for U.S. president is about 60%. Survey response rates are below 10%. Survey response is a much more optional thing, hence it makes sense to see much bigger swings in differential survey responses than in differential turnout. So, yes, differential turnout in voting is a thing, it’s just not as big as differential nonresponse in surveys.
2. I have no idea if Wolfers sent a tweet to me or whatever; all I know of this is from Hassan’s email to me.
Right. JW just tweeted, and copied Nate. I only sent Dr. Gelman the links to the tweets and their twitter discussion.
Apologies: in my comment about twitter, I was discussing nate silver, and for some reason wrote Gelman instead of Silver.
I refer interested readers to:
http://statmodeling.stat.columbia.edu/2016/08/05/dont-believe-the-bounce/ (second comment).
Let’s just say I don’t believe the “non-response” hypothesis but I do believe it’s testable.
There are panel polls that aim to alleviate the non-response swings to some degree, and they seem to show that actual preferences are changing. The notorious LA Times poll with the huge house effect in favour of Trump is one such — and its house effect is due to the overweighting of one 19-year-old black dude in the panel who says he’s a Trump voter.
The LA Times denies this claim (http://www.latimes.com/politics/la-na-pol-daybreak-poll-questions-20161013-snap-story.html). But of more interest to me, is, in your opinion, does the LA Times poll (leaving aside the bias) support or not support Gelman et. al.’s claim of non-response as driving variations in polling results? Your comment seems to indicate that the LA Times poll provides evidence against that hypothesis, but I’d like confirmation (if nothing else, so I can add in the link to that the next time the claims of non-response as the primary factor in polling variation come up).
It might be worth comparing the RAND ALP, which uses a similar methodology, but (IIRC, eventually) provides the full dataset to allow more robust analysis.
http://www.rand.org/labor/alp/2016-election-panel-survey.html
Of course, if you don’t pre-specify your hypothesis, once the data is released your claims lose a ton of credibility – so stake your claims here now!
There’s considerable evidence that partisan id fluctuates (see comment under numeric in) http://statmodeling.stat.columbia.edu/2016/08/05/dont-believe-the-bounce/ and Nate Silver (http://fivethirtyeight.com/features/election-update-is-the-presidential-race-tightening) indicates he’s agnostic (though Andrew has rejected other pollsters experience–“I’m willing to go with our evidence on this rather than trust the statements of Dick Morris or Mark Penn or Pat Cadell or whatever.” (Note these individuals have all run winning presidential campaigns, so maybe there shouldn’t be such a cursory rejection of their experience). I’m wondering if the non-response on the x-box is an artifact, such as the difference between live polling and robo-polling, which produces a large differential also.
Online panels seem to solve the differential response rates issue, but the people who do them might be more engaged:
Jeffrey A. Karp and Maarja Luhiste. Forthcoming. “Explaining Political Engagement with Online Panels: Comparing the British and American Election Studies.” in Public Opinion Quarterly
There is something amiss re this Silver/Wolfers spat. Wolfers is more than keen to refer to the betting market on Betfair when it suits him, and yet, on this occasion he consistently ignores the fact that the Betfair market has been more conservative re Clinton’s chances than Silver throughout the campaign (and still is). A case of having your cake and eating it.