Dan Kahan wrote:
You should do a blog on this.
I replied: I don’t like this article but I don’t really see the point in blogging on it. Why bother?
BECAUSE YOU REALLY NEVER HAVE EXPLAINED WHY. Gelman-Rubin criticque of BIC is *not* responsive; you have something in mind—tell us what, pls! Inquiring minds what to know.
Me: Wait, are you saying it’s not clear to you why I should hate that paper??
Certainly what say about “model selection” aspects of BIC in Gelman-Rubin don’t apply.
Me: OK, OK. . . . The paper is called, Bayesian Benefits for the Pragmatic Researcher, and it’s by some authors whom I like and respect, but I don’t like what they’re doing. Here’s their abstract:
The practical advantages of Bayesian inference are demonstrated here through two concrete examples. In the first example, we wish to learn about a criminal’s IQ: a problem of parameter estimation. In the second example, we wish to quantify and track support in favor of the null hypothesis that Adam Sandler movies are profitable regardless of their quality: a problem of hypothesis testing. The Bayesian approach unifies both problems within a coherent predictive framework, in which parameters and models that predict the data successfully receive a boost in plausibility, whereas parameters and models that predict poorly suffer a decline. Our examples demonstrate how Bayesian analyses can be more informative, more elegant, and more flexible than the orthodox methodology that remains dominant within the field of psychology.
And here’s what I don’t like:
Their first example is fine, it’s straightforward Bayesian inference with a linear model, it’s almost ok except that they include a bizarre uniform distribution as part of their prior. But here’s the part I really don’t like. After listing seven properties of the Bayesian posterior distribution, they write, “none of the statements above—not a single one—can be arrived at within the framework of orthodox methods.” That’s just wrong. In classical statistics, this sort of Bayesian inference falls into the category of “prediction.” We discuss this briefly in a footnote somewhere in BDA. Classical “predictive inference” is Bayesian inference conditional on hyperparameters, which is what’s being done in that example. A classical predictive interval is not the same thing as a classical confidence interval, and a classical unbiased prediction is not the same thing as a classical unbiased estimate. The key difference: when a classical statistician talks about “prediction,” this means that the true value of the unknown quantity (the “prediction”) is not being conditioned on. Don’t get me wrong, I think Bayesian inference is great; I just think it’s silly to say that these methods don’t exist with orthodox methods.
Their second example, I hate. It’s that horrible hypothesis testing thing. They write, “The South Park hypothesis (H0) posits that there is no correlation (ρ) between box-office success and “fresh” ratings—H0: ρ = 0.” OK, it’s a joke, I get that. But, within the context of the example, no. No. Nononononono. It makes no sense. The correlation is not zero. None of this makes any sense. It’s a misguided attempt to cram a problem into an inappropriate hypothesis testing framework.
I have a lot of respect for the authors of this paper. They’re smart and thoughtful people. In this case, though, I think they’re in a hopeless position.
I do agree with Kahan that the problem of adjudicating between scientific hypotheses is important. I just don’t think this is the right way to do it. If you want to adjudicate between scientific hypotheses, I prefer the approach of continuous model expansion: building a larger model that includes the separate models as separate cases. Forget Wald, Savage, etc., and start from scratch.