Jeremy Freese has the story.
To me, the sad thing is not that people who don’t understand statistics are doing research. After all, statistics is hard, and to require statistical understanding of all quantitative researchers would be impossible to enforce in any case. Indeed, if anything, one of the goals of the statistical profession is to develop tools such as regression analysis that can be used well even by people who don’t know what they are doing.
And the sad thing is not that the Proceedings of the National Academy of Science publishes weak work. After all, there’s a reason that PNAS is known as one of “the tabloids.”
To me, the sad thing is that certain researchers, after getting well-informed and open criticism, respond with defensive foolishness rather than recognizing that maybe—just maybe—some other people know more about statistics than they do.
Incompetence is not so bad—all of us are incompetent at times—but, as scientists, we should try to recognize the limitations of our competence.
It’s too bad because the himmicanes people always had an easy out: they could just say that their claims are purely speculative, that they’re drawing attention to a potentially important issue, that yes they did make some statistics errors but that’s not surprising given that they’re not statistics experts, and that they recommend that others follow up on this very important work. They could respond to the points of Freese and others that their claims are not convincing given huge variation and small sample size, by agreeing, and saying that they regret that their work was so highly publicized but that they think the topic is important. Their defensiveness is completely inappropriate.
I blame society
To return to one of our general themes of the past year or so, I think a key problem here is the discrete nature of the scientific publication process. The authors of the himmicanes/hurricanes paper appear to have the attitude, so natural among those of us who do science for a living, that peer-reviewed publication is a plateau or resting place: the idea is that acceptance in a journal—especially a highly selective journal such as PNAS—is difficult and is a major accomplishment, and that should be enough. Hence the irritation at carpers such as Freese who won’t let well enough alone. From the authors’ point of view, they’ve already done the work and had it approved, and it seems a bit like double jeopardy to get their work criticized in this way.
I understand this attitude, but I don’t have to like it.
P.S. Above image from classic Repo Man clip here.