Donny Williams writes:

I have a question I have been considering asking you for a while. The more I have learned about Bayesian methods, including regularly reading the journal Bayesian Analysis (preparing a submission here, actually!), etc., I have come to not only see that frequency properties are studied of Bayesian models, but it is the norm in more statistical papers that I have read.

As I am in psychology, why do you think Bayesian psychologists are so averse to this notion? It would seem that a reasonable expectation for new method (to the field at least) is for an evaluation under repeated use. Part of me thinks it is because they brought to psychology the approach of the mid 1900’s such as Jeffreys’ and, as a result, we are having decades-old debates in psychology. Furthermore, from reading more methods oriented journals, it is not so clear to me that much of the Bayes factor advocacy could have been published elsewhere, as it was more here is what we say Bayes can do but never demonstrated it, even when we know the truth (as in simulation).

Do you have any thoughts here?

My reply:

As a non-psychologist, I’m probably not the best person to opine on the psychology of psychologists’ choice of psychological methods. It does seem that some in psychology (and, before that, sociology) have rediscovered a naive enthusiasm for Bayes factors, which I do find frustrating. That said, just about any method can be useful, depending on how it’s used, so at some point you just have to do your best and move on.

Regarding the question of frequency properties of statistical methods: Yes, this is a good idea but you have to do it right. Over the past few decades I’ve seen a few zillion papers (and written a few of my own) featuring mind-numbing tables and graphs of frequency evaluations of statistical methods using simulation studies to show bias, variance, mean squared error, and coverage rates of various procedures. It’s often not clear what to do with such results, as any method should perform best when its assumptions are satisfied. This is not to say that simulation studies and frequency evaluations are useless, just that I think they often should be more focused. This paper for example features a very simple frequency analysis, not a big grid of simulations but I think it’s helpful in context.

> question of frequency properties of statistical methods: Yes, this is a good idea but you have to do it right.

I think this is almost always an elephant in the room when anyone presents a Bayesian analysis.

Based on my convenience sample informed biased posterior would be that roughly 70% will claim frequency properties are just not relevant, 20% that they are relevant but they are unsure how to assess them and 10% that they are relevant but they need to be more reasonable properties that are context specific than usual classical frequency properties such as uniform confidence interval coverage (NormalDeviate’s favorite property). See http://andrewgelman.com/2016/08/22/bayesian-inference-completely-solves-the-multiple-comparisons-problem/ and Case study 1 here http://andrewgelman.com/2017/04/19/representists-versus-propertyists-rabbitducks-good/

Hopefully that 10% will grow in the future, but I think that needs to be encouraged by more papers and talks making that very clear (especially to 70% that believes frequency properties are just not relevant).

Off-topic. I know precious little about Bayesian statistics, but I recognized the subject line. When I was a child, on the occasional nights when it was my father who read us a bedtime story, the odds-on choice was Damon Runyon.