Background: Hillary Clinton was given a 65% or 80% or 90% chance of winning the electoral college. She lost.
Naive view: The poll-based models and the prediction markets said Clinton would win, and she lost. The models are wrong!
Slightly sophisticated view: The predictions were probabilistic. 1-in-3 events happen a third of the time. 1-in-10 events happen a tenth of the time. Polls have nonsampling error. We know this, and the more thoughtful of the poll aggregators included this in their model, which is why they were giving probabilities in the range 65% to 90%, not, say, 98% or 99%.
More sophisticated view: Yes, the probability statements are not invalidated by the occurrence of a low-probability event. But we can learn from these low-probability outcomes. In the polling example, yes an error of 2% is within what one might expect from nonsampling error in national poll aggregates, but the point is that nonsampling error has a reason: it’s not just random. In this case it seems to have arisen from a combination of differential nonresponse, unexpected changes in turnout, and some sloppy modeling choices. It makes sense to try to understand this, not to just say that random things happen and leave it at that.
This also came up in our discussions of betting markets’ failure in Trump in the Republican primaries, Leicester City, and Brexit. Dan Goldstein correctly wrote that “Prediction markets have to occasionally ‘get it wrong’ to be calibrated,” but, once we recognize this, we should also, if possible, do what the plane-crash investigators do: open up the “black box” and try to figure out what went wrong that could’ve been anticipated.
Hindsight gets a bad name but we can learn from our failures and even from our successes—if we look with a critical eye and get inside the details of our forecasts rather than just staring at probabilities.