Skip to content

Evaluating predictions of political events

Mike Cohen writes:

The recent events in Egypt raise an interesting statistical question. It is of course common for news stations like CNN to interview various officials and policy experts to find out what is likely to happen next. The obvious response of people like us is why ask such people when they didn’t foresee a month ago that these dynamic events were about to happen. One would instead like to hear from those experts that did predict that something was about to happen in Tunisia, and Egypt, and Jordan, and maybe Yemen, etc. Well, are there such people? My friend Bob Burton says that of course one can find such people in the sense that they made such predictions, but that is like finding counties that have voted for the President in the last five elections, big deal, or psychics that predicted the last assassination, again big deal. There is a good deal of truth in that. However, it seems like we do a little better. There are two points to make. First, there is an information theory aspect to how impressed we should be by predictions that are made. In the following I am glossing over a very important point, which is how specific the predictions are, and that is worth thinking a lot about. But speaking loosely, for a dichotomous question with both sides close to equal a priori, making a prediction that turns out to be correct is no big deal. But predicting the date and manner in which some political event would occur, such as a change in government well in advance, when the general a priori view was that nothing was going on, would be extremely impressive. So question one is can one measure how impressive a prediction is, to help us understand who our experts really are, and what experts are just guessing well? Second, does Efron’s work 40 years ago on assessing whether Bode’s Law has a physical basis relate to this question? Maybe not, but Efron was looking to see whether the closeness of a hypothesis and data was such that one could say that it was not likely to have occurred by chance, and there seems to be something similar going on when comparing a number of such predictions by someone and their closeness to later events. Any thoughts?

I replied as follows:

My quick thought is that any expert forecasts should be evaluated not in a vacuum but in the context of their qualitative models. Regarding Egypt, you might be interested in Daniel Drezner’s remarks about political scientists’ work in that area.

To which Cohen pointed out that an expert could predict that something was going to happen without having an explicit theory that they were operating from and that would still be of interest, though less so, than if they had put forward how they arrived at such predictions.


  1. jflycn says:

    It's not just political science. It's about all macro-level social sciences.
    Statistics tells us we can make some predictions. On the other side, it also tells us about unpredictable nature of the world. Most people always forget the second point.

    Check this:
    Stephen Jay Gould and the Contingent Nature of History

  2. Dave Robinson says:

    Personnaly, I'd like to see this implemented in the US for those involved with political predictions.

    Romanian witches could face jail if predictions don't come true. A month after Romanian authorities began taxing them for their trade, the country's soothsayers and fortune tellers are cursing a new bill that threatens fines or even prison if their predictions don't come true.

  3. statc says:

    I was wondering if you are going to post something on Egypt, since it is getting all the attentions in the news these days.

  4. Saul Radshaw says:

    how do we evaluate conjecture, when conjecture is still such a loose science (esp. w/ history and poly sci)? related fields train us to study the past and ongoing, which one would think lends itself to some understanding of the future and human behavior on a larger scale; it does not. Neither does any social science, conjecture should be a school on it's own, drawing from everything, predictions experts make are always based on their narrow range of expertise, and assume their range is vital to current events (when one category in particular only plays a small role). You would need a lot of people feeding expert info to a generalist (like the pseudo-scientists of yore) to process and extrapolate for all the given info in an area.