A correspondent writes:
There is now a pre-registration site for parapsychology:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4332280/
Registered StudiesThere are several experiments that completely flopped, some that haven’t been published after a few years (like Daryl Bem and collaborator’s attempted large-n replication of Bem), and some that report positive pre-registered results (but probably usually have repeatable methodological problems).
Sorry but there’s no way this can work. Here’s the problem: A researcher such as Bem plans a study. He then gets one of his associates such as Larry Sabato (see above) to see into the future to find out which of many predictors will show a strong correlation with the outcome variable. Then the researcher can cheat and write the pre-registration plan to be consistent with what will happen. Then the data are gathered . . . p less than 0.05 . . . but it’s just data dredging. All you need is a good crystal ball and the pre-registration is defeated.
Damn! Another seemingly-promising idea that can’t work.
I believe there is a similar problem in placebo research https://xkcd.com/1526/
A fun experiment for Bem would be to check whether Sabato and his ball can predict which psych studies reproduce better than p-values do. Call it the “Reductio ad absurdum of the last century of statistics” experiment.
Some researchers have begun to raise the possibility that pre-registration could be faked. Collect and analyze the data beforehand, “pre-register” the study, bam. So the story you tell might be more likely than anyone thinks.
Registered reports should be reviewed, and reviewers encouraged to suggest insignificant but detectable changes to the design. It would help prevent this kind of shenanigans, and also encourage theoretical robustness.
Most trial registries allow retrospective editing of the entry. I’ve reviewed papers where I’ve pointed out the inconsistency of an RCT with the registered protocol, and the author simply answer “Oh thanks for that, I’ll change the trial registry entry to make it more consistent”.
I think the WHO trial registry actually highlights changes, which is massively helpful to track down sneaky people.
“Most trial registries allow retrospective editing of the entry. I’ve reviewed papers where I’ve pointed out the inconsistency of an RCT with the registered protocol, and the author simply answer “Oh thanks for that, I’ll change the trial registry entry to make it more consistent”.”
Aargh!
This one is better than that, since entries are irreversible commitments by the authors.
Couldn’t this be spotted somehow?
For instance, if the data collected has a time/date entry this could be compared to the pre-registration. Let’s see who will be the 1st to get busted faking pre-registrations. Let’s hope it will be considered as fraud and that they will be sacked. Good riddance!
James Randi has been offering money based on a kind of pre-registered ‘paranormal’ test for decades now. https://en.wikipedia.org/wiki/One_Million_Dollar_Paranormal_Challenge
The joke in the title reminds me of John Sladek’s line (paraphrasing) that John Schliffendoerffer of the Institute for Paranormal Studies found convincing evidence of precognition in a study that asked volunteers to predict whether toast would fall butter side up or butter side down. “The study itself failed, but eerily I *knew* it was going to fail ahead of time.”