David Hogg points me to this discussion:
Martin Strasbourg and I [Hogg] discussed his project to detect new satellites of M31 in the PAndAS survey. He can construct a likelihood ratio (possibly even a marginalized likelihood ratio) at every position in the M31 imaging, between the best-fit satellite-plus-background model and the best nothing-plus-background model. He can make a two-dimensional map of these likelihood ratios and show a the histogram of them. Looking at this histogram, which has a tail to very large ratios, he asked me, where should I put my cut? That is, at what likelihood ratio does a candidate deserve follow-up? Here’s my unsatisfying answer:
To a statistician, the distribution of likelihood ratios is interesting and valuable to study. To an astronomer, it is uninteresting. You don’t want to know the distribution of likelihoods, you want to find satellites . . .
I wrote that I think this makes sense and that it would actualy be an interesting and useful research project to formalize this as a decision problem. I’ve seen this sort of question arise in genetics (where should the p-value threshold be when you’re selecting N out of a million genes) but it’s frustrating because the cost-benefit calculations always seem implicit. I’d like to see it done out in the open.
I agree! In one—just one—of my papers we put an explicit utility model: http://arxiv.org/abs/0910.2233
The utility model is on page 17 and we use it explicitly on page 18 and on. It is a web system (up now at http://nova.astrometry.net/ and running inside flickr.com ) that has to *decide* whether to return results to the user or not, given probabilistic information about a submitted image.