In a comment to this entry on using (y+1)/(n+2) instead of y/n, Aleks writes, “Instead of debating estimates, why not debate priors?” My quick answer to this is, yeah, sure, but there can be information in “n” also. Which formally means that the prior distribution for theta will depend on n, but sometimes this can be more conveniently considered in terms of the properties of estimates.
To look at it another way, my entry was linked to here by “RPM” and elicited the following comment from Jan Moren:
So the chance of me having a three-way with Madeleine Albright and an 17-tentacled alien from Arcturus humming show tunes on the roof of Sogo department store in downtown Osaka is 0.5? Sweet!
This isn’t right, though, since there’s no “y” or “n” in this Madeline Albright scenario. Jan is commenting on the implied prior distribution, not on the estimate.
I see Jan Moren's comment as a reminder that we should really be thinking in terms of priors not in terms of estimates. Summarizing a "all probabilities are equally likely" prior into a 0.5 estimate is the problem, not the vague prior.
It has happened to me several times that I came up with a prior that would depend on n, so it couldn't be expressed as a Bayesian prior. I want to think of ways of getting away with that: in some cases it might be practical (for efficiency reasons) to open up additional levels of complexity with a sufficient sample size.
Aleks,
I don't know why you say that a prior that depends on n is not Bayesian. It's just a joint prior on (n,theta), which is perfectly kosher. For example, I wouldn't expect to see a n=10 study of a phenomenon with p=1E-6.
There is too a y and an n: y is zero, and n is zero, so the estimate is 1/2.
Or you could say, assuming Jan Moren reached puberty twenty years ago, that y=0 and n=1–in which case the estimate is 1/3.
OK, OK, let me try again. First off, I can accept that y=0 for Jan Moren but I don't see what n is. Perhaps n=1, or n=1 billion (an order-of-magnitude estimate of the number of seconds in Jan's life so far), whatever. It's just not a clearly defined experiment.
More to the point, for any estimate (or, for that matter, prior distribution), you can come up with examples where it won't make sense. On one hand, the (y+1)/(n+2) estimate doesn't work well for Jan's example. On the other hand, the y/n estimate doesn't work well for examples where n is 10 or 20 and p is between .05 and .95. My claim is that these estimates are much more likely to be used in my scenarios than in Jan's scenario. It's not that extremely low probabilities don't happen, it's just that you wouldn't usually try to estimate them using small-n studies of this sort.
More generally, I estimate very low probabilities using a combination of empirical data and modeling of precursor events; see here for our paper on estimating the probability of events that have never occurred.