A correspondent writes:
I thought you might enjoy this…
I’m refereeing a paper which basically looks at whether survey responses on a particular topic vary when the question is asked in two different ways. In the main results table they split the sample along several relevant dimensions (education; marital status; religion; etc). I give them credit for showing all the results, but only one differential is statistically significant at 5%, and of course they focus the interpretation on that one. In my initial report, I asked if they either had tried or would try correcting for multiple hypothesis testing. I just received their response:
“We agree with the referee, but we do not think it is possible given that we really do not have enough power.”
So they left it as is and don’t discuss the issue at all in the revision!
As is often the case, this is an example where I suspect we’d be better off had p-values never been invented.