The other day I was talking with someone who knows Daryl Bem a bit, and he was sharing his thoughts on that notorious ESP paper that was published in a leading journal in the field but then was mocked, shot down, and was repeatedly replicated with no success. My friend said that overall the Bem paper had positive effects in forcing psychologists to think more carefully about what sorts of research results should or should not be published in top journals, the role of replications, and other things.
I expressed agreement and shared my thought that, at some level, I don’t think Bem himself fully believes his ESP effects are real. Why do I say this? Because he seemed oddly content to publish results that were not quite conclusive. He ran a bunch of experiments, looked at the data, and computed some post-hoc p-values in the .01 to .05 range. If he really were confident that the phenomenon was real (that is, that the results would apply to new data), then he could’ve easily run the experiments on a bunch more students, gathering enough data so that nobody could doubt his claims. But Bem didn’t do that. Instead, once he felt he’d reached the statistical significance plateau, he stopped and submitted to the journal. This behavior is consistent with the idea that he did not want to push his claims further, instead wanting to get into print before any new data could reveal problems with his study.
Ironically, I told my friend, Bem’s strategy didn’t work. Yes, the paper was published in a top journal. But, rather than this publication making the result more plausible, the reverse happened: the implausible claims reduced the perceived validity of psychology studies more generally. The journal didn’t establish the truth of the finding; instead, the finding dragged the journal down.
My friend then unleashed an amazing theory: that Bem really really doesn’t believe these ESP claims, that he did this whole project with a straight face to demonstrate problems with our current system of statistical/scientific research and publishing. Never breaking character, Bem will take this secret to his grave.
I don’t know, but my friend is the one who knows Bem, and that’s what he tells me.