Philip Jones writes:
As an interested reader of your blog, I wondered if you might consider a blog entry sometime on the following question I posed on CrossValidated (StackExchange).
I originally posed the question based on my uncertainty about 95% CIs: “Are all values within the 95% CI equally likely (probable), or are the values at the “tails” of the 95% CI less likely than those in the middle of the CI closer to the point estimate?”
I posed this question based on discordant information I found at a couple of different web sources (I posted these sources in the body of the question).
I received some interesting replies, and the replies were not unanimous, in fact there is some serious disagreement there! After seeing this disagreement, I naturally thought of you, and whether you might be able to clear this up.
Please note I am not referring to credible intervals, but rather to the common medical journal reporting standard of confidence intervals.
First off, I’m going to forget about the official statistics-textbook interpretation, in which a 95% confidence interval is defined as a procedure that has a 95% chance of covering the true value. For most of the examples I’ve ever seen, this interpretation is pretty useless because the goal is to learn about the situation we have right now in front of us, not merely to make a statement with certain average properties.
I would say that the usual interpretation of a confidence interval is as a set of parameter values that are consistent with the data. Typically the values near the center of the interval are more consistent, and sometimes this idea is formalized by thinking about hypothetical nested 1%, 2%, 3%, …, 99% intervals, where the more central parameter values are in more of these intervals.
The real problem is that the interval will exclude the true value at least 5% of the time. 5% doesn’t sound like much, but given that it is the more significant findings that get noticed, these can be an important 5%. Also, when the sample size is small, the confidence interval can include lots of implausible values too. Consider the notorious claim that beautiful parents were more likely to have girls. Here, the confidence interval included all sorts of big numbers (for example, the data were consistent with beautiful parents being 10 percentage points more likely to have a girl, compared to ugly parents) that a quick literature review revealed were highly implausible. This was a setting where the prior information was much stronger than the data.