Chris Chambers pointed me to a blog by someone called Neuroskeptic who suggested that I preregister my political science studies:
So when Andrew Gelman (let’s say) is going to start using a new approach, he goes on Twitter, or on his blog, and posts a bare-bones summary of what he’s going to do. Then he does it. If he finds something interesting, he writes it up as a paper, citing that tweet or post as his preregistration. . . .
I think this approach has some benefits but doesn’t really address the issues of preregistration that concern me—but I’d like to spend an entire blog post explaining why. I have two key points:
1. If your study is crap, preregistration might fix it. Preregistration is fine—indeed, the wide acceptance of preregistration might well motivate researchers to not do so many crap studies—but it doesn’t solve fundamental problems of experimental design.
2. “Preregistration” seems to mean different things in different scenarios:
A. When the concern is multiple comparisons (as most notoriously in that Bem ESP study), “preregistration” would require completely specifying—before the data are collected—the experimental design, data collection protocols, data exclusion rules, and selecting ahead of time the comparisons to be performed and the details of the statistical analysis.
B. When the concern is the file drawer (the idea that Gelman or some other researcher performs lots of studies, and we want to hear about the failures as well as successes), “preregistration” is a “bare bones summary” that can be tweeted.
I think we can all agree that Preregistration A and Preregistration B are two different things. There’s no way I could specify all my data processing and analysis decisions for a study, in only 140 characters!
Neither my point 1 or point 2 is an argument against preregistration! Sure, preregistration doesn’t solve problems of experimental design. It doesn’t cure cancer either, that doesn’t mean it’s useless. But I think it’s important to be clear what sort of preregistration we’re talking about in any particular case, and what we expect preregistration to be doing.
1. Preregistration is fine but it doesn’t solve your problem if studies are sloppy, variation is high, and effects are small—especially if you try to study within-person effects using between-person designs
Neuroskeptic has a more recent post expressing a noncommittal view of the claim that hormonal changes during the menstrual cycle impact political and religious beliefs. Given that political science is (I assume) not among Neuroskeptic’s areas of expertise, I respect his or her decision to hold off judgment on the issue. However, given that this is an area of my expertise, I can assure Neuroskeptic that the original published claim on this topic is indeed (a) unsupported by the researchers’ data, because of reasons of multiple comparisons (see here) for much discussion, and (b) extremely implausible given what we know about the stability of public opinion, especially in recent elections. Of this study and a recent nonreplication, Neuroskeptic writes:
I know of only one way to put a stop to all this uncertainty: preregistration of studies of all kinds. It won’t quell existing worries, but it will help to prevent new ones, and eventually the truth will out.
I think this is overstated and way too optimistic. In the case of the ovulation-and-voting study, the authors have large measurement error, high levels of variation, and they’re studying small effects. And all this is made even worse because they are studying within-person effects using a between-person design. So any statistically significant difference they find is likely to be in the wrong direction and is essentially certain to be a huge overestimate. That is, the design has a high Type S error rate and a high Type M error rate.
2. Should I preregister my research?
As I wrote above, the question is not just if I should preregister but also how I should do it.
I do think that blogging my research directions ahead of time (or archiving time-stamped statements, if I want to let my ideas develop in secret for awhile) would be a good idea. It would give people a sense of what worked, what didn’t work, and what I’m still working on. In many ways I do a lot of this already—blogging my half-formed thoughts which, years later, turn into papers—but I can see the argument for doing this more formally.
But this sort of blogging would not address multiple comparisons problems. For example, if Bem had tweeted ahead of time that he was doing an ESP study and focusing on prediction of sexual images, or if Beall and Tracy had tweeted ahead of time that hey were investigating the hypothesis that that fertile women were more likely to wear red, no, in neither cases would that have been enough. In both cases, there were just too many data-processing and data-analysis choices that would not fit in that research plan.
So, what about the full preregistration? Not what goes in a tweet but a complete step-by-step plan of how I would choose, code, and analyze the data? That might have been a good idea for Bem, Beall, and Tracy, or maybe not, but it would certainly not work for me in my political science research (or, for that matter, in my research in statistical methods). Many of my most important applied results were interactions that my colleagues and I noticed only after spending a lot of time with our data.
Just to be clear: you can see that I’m not disagreeing with Neuroskeptic. He or she suggested that I do tweet-like announcements of my research plans, and I agree that’s a good idea. It’s just not the same as preregistration, at least not as commonly understood.
P.S. This discussion published last year is relevant too.