David Radwin writes:
I am seeking a statistic measuring an estimate’s reliability or stability as an alternative to the coefficient of variation (CV), also known as the relative standard error. The CV is the standard error of an estimate (proportion, mean, regression coefficient, etc.) divided by the estimate itself, usually expressed as a percentage. For example, if a survey finds 15% unemployment with a 6% standard error, the CV is .06/.15 = .4 = 40%.
Some US government agencies flag or suppress as unreliable any estimate with a CV over a certain threshold such as 30% or 50%. But this standard can be arbitrary (for example, 85% employment would have a much lower CV of .06/.85 = 7%), and the CV has other drawbacks I won’t elaborate here. I don’t need an evaluation of the wisdom of using the CV or anything else for measuring an estimate’s stability, but one of my projects calls for such a measure and I would like to find a better alternative.
Can you or your blog readers suggest a different measure of reliability?
My reply: If you are stuck here, go back to first principles. If you just need a measure, you can supply both the estimate and the standard error. But it sounds like you are looking for a rule of some sort? Maybe it would help to try to quantify the gains and losses from classifying an estimate as “stable.” Then you could do a decision analysis. I’m not saying that the formal decision analysis should decide your answer but it could give you some intuition about what various proposed procedures are doing.
Based on my experiences, I think you could make general progress by constructing a solution to your specific problem.
Any other thoughts?