Gabriel Power asks the above question, writing:
I don’t recall seeing, on your blog or elsewhere, this question raised directly. Of course there is much talk about the importance of replication, mostly by statisticians, and economists are grudgingly following suit with top journals requiring datasets and code.
But why not make it a simple requirement? No replication, no publication.
I suppose that it would be too time-consuming (many reviewers shirk even that basic duty) and that there is a risk of theft of intellectual property.
My reply: In this context, “replication” can mean two things. The first meaning is that the authors supply enough information that the exact analysis can be replicated (this information would include raw data (suitably anonymized if necessary), survey forms, data collection protocols, computer programs and scripts, etc. Some journals already do require this; for example, we had to do it for our paper in the Quarterly Journal of Political Science. The second meaning of “replication” is that the authors would actually have to replicate their study, ideally with a preregistered design, as in the “50 shades of gray” paper. This second sort of replication is great when it can be done, but it’s not in general so easy in fields such as political science or economics where we work with historical data.