Changhe Yuan writes,

I’m working on the problem of finding explanations for new observations in Bayesian networks. I’m considering using the Bayes factor (BF) to rank different hypotheses, because it has the capability to compare hypotheses with different sets of variables. However, I have the following example that apparently makes BF less desirable.

We have two variables, A and B. B is observed to be b. We have the following distributions:

P(a1)=1/(10^100), P(a2)=1-1/(10^100);
P(b|a1)=.9, P(b|a2)=.8;

So a1 is extremely unlikely, but b is slightly more likely when given a1 than a2. If we calculate BF for the two hypotheses A=a1 or A=a2, we get

BF(a1)=P(b|a1)/P(b|a2)=9/8,
BF(a2)=P(b|a2)/P(b|a1)=8/9.

So it says that a1 is a better explanation than a2 even though a1 is basically impossible, and it has only a slightly stronger conditional relationship with b.

In comparison, the posterior probability distribution of A will say that a2 is a wildly better explanation.

Does this example show that BF selects a bad explanation?

My response: The Bayes factor is the likelihood ratio and tells you the information in the data. You need to combine with info in the prior distribution to get the posterior probabilities of what you want.

Usually I don’t like model selection anyway (I prefer continuous model expansion, as discussed in the relevant section of Chapter 6 of Bayesian Data Analysis), but if you’re working with the Bayes factor, you do want to include the prior probabilities.