John Skilling wrote this response to my discussion and rejoinder on objections to Bayesian statistics.
John’s claim is that Bayesian inference is not only a good idea but also is necessary. He justifies Bayesian inference using the logical reasoning of Richard Cox (1946), which I believe is equivalent to the von Neumann and Morgenstern (1948) derivation that those of us in the social sciences are familiar with.
I have no objection to the Cox/Neumann/Morgenstern argument (which is also associated with Keynes and others). However, all our models have flaws, so the big step that all of us have to make when recommending Bayesian methods is to believe that they work well even with the imperfect models that we use in practice. Models that in social science are surely far more imperfect than those used in physics (which may be one reason that physicist Skilling is so accepting of Bayes). Skilling cites Jaynes, whose writings have been a major influence in my own conception of Bayesian statistics. (The first scientific conference I ever attended, I think, was Skilling’s Jaynes-inspired maximum entropy conference in 1988.) In particular, I got from Jaynes the idea that one should take any model, even a simple model, seriously enough to try to understand where it doesn’t fit reality and how it should be improved.
In conclusion: I think Skilling’s normative argument is powerful, but not definitive on its own, because we want statistical methods to work well even when models have serious flaws. That’s why I take a more pluralistic approach. (But, as I explain in my rejoinder (linked above), I am not at all convinced by arguments about classical unbiasedness or coverage.)
To put it another way, the classical ideas of sufficiency, ancillarity, confidence coverage, hypothesis testing, etc etc: I’m happy to trash all these. But there is another set of principles out there, based on external validation (sometimes approximated by cross-validation), that seems valid to me, and does not necessarily rely on Bayes.
P.S. See also Larry Wasserman’s further comments.
Yeah, so John Norton will probably have something to say about the claim that "we MUST use Bayes" on pain of logical contradiction.
Andrew, I'd be interested in any thoughts you might have about the critiques contained in his "Challenges to Bayesian Confirmation Theory" paper (though they seem somewhat tangential to your concerns).
Oh, dear, Norton brings up Glymour's argument "problem of old evidence" . Glymour's argument is bogus. It depends on the erroneous notion that if evidence x has been observed as old evidence, then Pr(x), which Skilling is calling "evidence" and statisticians call "the prior predictive" or "the marginal likelihood," is equal to 1. [Glymour's argument was that Pr(x)=1 so Pr(x|T)=1 for theory T, therefore Pr(T|x)=Pr(x|T)Pr(T)/Pr(x) therefore Pr(T|x)=Pr(T), the posterior is equal to the prior and you haven't learned anything. This is a nonsense argument, since Pr(x) is just a calculable function of the theory and the priors, and can't be evaluated by Glymour's naive notion that evidence, once observed, has marginal likelihood 1.]
Norton's take on it is that once you know the evidence, it becomes part of the background information and can't be removed. This is also nonsense. Clearly, in the case of Einstein and general relativity, no one had any notion that x, in this case the perihelion advance of Mercury, would have an explanation in terms of general covariance as posited in general relativity. It just fell out of the theory (and by the way, there's doubt that Einstein was even aware of the perihelion advance of Mercury). Thus, knowledge of x, even though already known, does not affect the calculation. The likelihood ratio is computed from the theory…say, the likelihood ratio Pr(x|T)/Pr(x|T') where T' is a second theory, and this ratio is NOT in general equal to 1, since it is a function only of the models and the particular x observed. Glymour would have it that Pr(x|T)/Pr(x|T')=1, but that's wrong. But the likelihood ratio being in general not equal to 1, and calculable from the theory in a way that is independent of whether x is known or not, this shows that evidence x, through the likelihood ratio, can indeed support one theory against another, even if x is already observed.
What Glymour actually proved is that a Bayesian can't use the same piece of evidence twice. But that's a completely different issue. Unfortunately, Glymour's notation was botched, so he wasn't able to see what he actually proved.
When I read a philosopher philosophizing on Bayesianism, the first thing I look for is things like whether they take Glymour's argument seriously, or some others like Hempel's Raven "paradox", which Jack Good disposed of many years ago. Generally, if the philosopher takes these seriously, I don't take the philosopher seriously.
Oops, forgot to sign my comment ("Anonymous" above)
Somehow the "Remember Me" box got unchecked when I had to force quit Firefox.
"But there is another set of principles out there, based on external validation (sometimes approximated by cross-validation), that seems valid to me, and does not necessarily rely on Bayes."
Some links or references on this would be appreciated.
Keith
Has Norton, Glymour, etc ever responded to that argument? If the flaw is that straightforward, it would seem like some proponent of "the problem of old evidence" would confront it. I am always suprises me how much disagreement there is about such things.
Keith,
I have no particular reference, but I'd say: most of econometrics, much of machine learning, and some of biostatistics (e.g., generalized estimating equations). I'm thinking of methods that explicitly are set up to have good properties either with minimal assumptions or allowing for the assumptions to be off.
Roger Rosenkrantz criticised Glymour's argument in the same volume that Glymour presented his paper, along the same lines as I did. I've not seen any rebuttal. This was decades ago.