Biostatistician Jeff Leek writes:
Think about this headline: “Hospital checklist cut infections, saved lives.” I [Leek] am a pretty skeptical person, so I’m a little surprised that a checklist could really save lives. I say the odds of this being true are 1 in 4.
I’m actually surprised that he’s surprised, since over the years I’ve heard about the benefits of checklists in various arenas, including hospital care. In particular, there was this article by Atul Gawande from a few years back. I mean, sure, I could imagine that checklists might hurt: after all, it takes some time and effort to put together the checklist and to use it, and perhaps the very existence of the checklist could give hospital staff a false feeling of security, which would ultimately cost lives. But my first guess would be that people still don’t do enough checklisting, and that the probability is greater than 1/4 that a checklist in a hospital will save lives.
Later on, Leek writes:
Let’s try another headline: “How using Facebook could increase your risk of cancer.” Without looking at the study, I’d probably think “no way.” To my mind, the odds that this is right may be something like 1 in 10.
Whoa. He’s saying that his prior probability of this happening is as high as 1/10? That’s only 1/2.5 his prior probability that a checklist will save lives.
Here we can see one of the problems with subjective priors. It’s hard to get the scale right. I’m reminded of what George Orwell wrote about book reviewing: if you review 10 books a week, and if your scale is such that Hamlet is a good play and Great Expectations is a good read, how to you calibrate all the material of varying quality that you are sent to review? The only answer is that books are reviewed relative to expectations, and you can’t say that the latest bestseller is crap just cos it doesn’t live up to the standards of Shakespeare.
Similarly, I have a feeling that Leek is setting his priors relative to expectations. In his first example, sure, we have a general belief that checklists are important, but Leek compresses his scale by invoking a general skepticism. So, instead of saying that checklists probably work, he dials down his probability to 1/4. Now to the second example. Of course using Facebook does not give you cancer. But we can’t just set the probability to 0. Indeed, even thinking about the question implies that the probability is nonzero, and then we get to thinking: hmm, you use Facebook and you stay indoors more, then maybe you don’t get enough exercise or not enough vitamin C . . . ok, maybe it’s possible. And that gets you to the probability of 1/10.
But this 1/10 is not on the same scale as the earlier 1/4. The 1/4 referred to the probability that a checklist really works to save lives, whereas the 1/10 is the probability that there’s something, somewhere associated with Facebook use that is also associated with cancer risk in some small way (as there’s no realistic way this effect could be large).
This illustrates a general problem, not just with priors and Bayesian statistics but with scientific measurement in general. It’s hard to talk about probabilities, or any other numerical measurements, without some definition of what is being measured. (As regular readers of this blog know, similar problems arise with p-values.)
I respect that Leek was writing a general-interest article on a news website and so he had to simplify. My point is not to pick on him but rather to bring some attention to the general problems of probability assignment. It’s easy to say that we think a treatment works or doesn’t work, or that a certain pattern is real or not, but when we start to assign probabilities, I think we need to think more carefully about what are the events we are referring to.