I’ve been writing a lot about my philosophy of Bayesian statistics and how it fits into Popper’s ideas about falsification and Kuhn’s ideas about scientific revolutions.
Here’s my long, somewhat technical paper with Cosma Shalizi.
Here’s our shorter overview for the volume on the philosophy of social science.
Here’s my latest try (for an online symposium), focusing on the key issues.
I’m pretty happy with my approach–the familiar idea that Bayesian data analysis iterates the three steps of model building, inference, and model checking–but it does have some unresolved (maybe unresolvable) problems. Here are a couple mentioned in the third of the above links.
Consider a simple model with independent data y_1, y_2, .., y_10 ~ N(θ,σ^2), with a prior distribution θ ~ N(0,10^2) and σ known and taking on some value of approximately 10. Inference about μ is straightforward, as is model checking, whether based on graphs or numerical summaries such as the sample variance and skewness.
But now suppose we consider μ as a random variable defined on the integers. Thus θ = 0 or 1 or 2 or 3 or … or -1 or -2 or -3 or …, and with a discrete prior distribution formed by the discrete approximation to the N(0,10^2) distribution. In practice, with the sample size and parameters as defined above, the inferences are essentially unchanged from the continuous case, as we have defined θ on a suitably tight grid.
But from my philosophical position, the discrete model is completely different: I have already written that I do not like to choose or average over a discrete set of models. This is a silly example but it illustrates a hole in my philosophical foundations: when am I allowed to do normal Bayesian inference about a parameter θ in a model, and when do I consider θ to be indexing a class of models, in which case I consider posterior inference about θ to be an illegitimate bit of induction? I understand the distinction in extreme cases–they correspond to the difference between normal science and potential scientific revolutions–but the demarcation does not cleanly align with whether a model is discrete or continuous.
Another incoherence in Bayesian data analysis, as I practice it, arises after a model check. Judgment is required to decide what to do after learning that an aspect of data is not fitted well by the model–or, for that matter, in deciding what to do in the other case, when a test does not reject. In either case, we must think about the purposes of our modeling and our available resources for data collection and computation. I am deductively Bayesian when performing inference and checking within a model, but I must go outside this framework when making decisions about whether and how to alter my model.
In my defense, I see comparable incoherence in all other statistical philosophies:
– Subjective Bayesianism appears fully coherent but falls apart when you examine the assumption that your prior distribution can completely reflect prior knowledge. This can’t be, even setting aside that actual prior distributions tend to be chosen from convenient parametric families. If you could really express your uncertainty as a prior distribution, then you could just as well observe data and directly write your subjective posterior distribution, and there would be no need for statistical analysis at all.
– Classical parametric statistics disallows probabilistic prior information but assumes the likelihood function to be precisely known, which can’t make sense except in some very special cases. Robust analysis attempts to account for uncertainty about model specification but relies on additional assumptions such as independence.
– Classical nonparametric methods rely strongly on symmetry, translation invariance, independence, and other generally unrealistic assumptions.
My point here is not to say that my preferred methods are better than others but rather to couple my admission of philosophical incoherence with a reminder that there is no available coherent alternative.
To put it another way, how would an “AI” do Bayesian analysis (or statistical inference in general)? The straight-up subjective Bayes approach requires the AI to already have all possible models specified in its database with appropriate prior probabilities. That doesn’t seem plausible to me. But my approach requires models to be generated on the fly (in response to earlier model checks and the appearance of new data). It’s clear enough how an AI could perform inference on a specified graphical model (or on a mixture of such models); it’s not so clear how an AI could do model checking. When I do the three steps of Bayesian data analysis, human input is needed to interpret graphs and decide on model improvements. But we can’t ask the AI to delegate such tasks to a homunculus. Another way of saying this is that, if Bayesian data analysis is a form of applied statistics, an AI can’t fully do statistics until it can do science. In the meantime, though, the computer is a hugely useful tool in each of the three stages of Bayesian data analysis, even if it can’t put it all together yet. I’m hoping that by automating some of the steps required to evaluate and compare models, we can get a better sense of what outside knowledge we are adding at each step.
P.S. Cosma Shalizi writes:
If your graphical model does not have all possible edges, then there are conditional independencies which could be checked mechanically (and there are programs which do things like that). I know you don’t like zeroes in the model, but it’s at least a step in the right direction, no?
To which I reply:
Yup. Or at times it could be a step in the wrong direction, depending on the model. But if the program has the ability to check the model (or at least to pass the relevant graphs on to the homunculus), then I would think that working with conditional independence approximations could be a useful way to move forward.
That’s part of my incoherence. On one hand, I hate discrete model averaging. On the other hand, I typically end up with one model, which is just a special case of an average of models.