Skip to content
 

Write your congressmember to require researchers to publicly post their code?

Stephen Cranney writes:

For the past couple of years I have had an ongoing question/concern . . . In my fields (sociology and demography) much if not most of the published research is based on publicly available datasets; consequently, replicability is literally a simple matter of sending or uploading a few kilobytes of code text. (Of course, getting somebody’s code is like pulling teeth; people often conveniently lose it after the fact, can’t track down the coauthor who wrote the code, etc.)

I understand that replicating experimental results can be onerous (which doesn’t mean we shouldn’t do it!), but it seems that as replicating analyses done on survey and other non-experimental data is exponentially simpler, it represents a kind of low-hanging fruit in the replication crisis. However, I haven’t seen a lot of efforts to address this type of replication.

I know that some economics journals require researchers to submit the code they used to derive their results, but I haven’t seen any kind of movement to disseminate this practice more widely (in theory, with nothing to hide, it should be quite painless for the researchers). Also, has anybody raised the specter of legislation mandating that federally funded research publicly post their code (again, in theory something that should be painless)? I would definitely get involved in that write-your-congressperson movement if it developed any steam.

My response: I have no strong opinion on whether such legislation would be a good thing, but I do agree that duplication (re-running code) should be easier than replication (gathering new data). Also, amazingly enough, even something as simple as duplication on a new available dataset can be a challenge; see here for an example (with Rayleigh and Yair).

Relatedly, Tom Daula (whom we last encountered in the context of the $100,000 time series challenge) writes:

I wrote you before about “The Replication Network”, but they announced a new journal that will publish all replication studies. From the website:

Furthermore, [International Journal for Re-Views in Empirical Economics] IREE publishes synthesizing reviews, micro-data sets and descriptions thereof, as well as articles dealing with replication methods and the development of standards for replications…To make replication a little less thankless, all publications in IREE are citable. Each article, data set, and computer program is assigned a DOI. In addition, data sets are stored in a permanent repository, the ZBW Journal Data Archive. This provides a platform for authors to gain credit for their replication-related research.

11 Comments

  1. Dale Lehman says:

    Regarding the new journal: I have a piece I was considering sending to another journal which explicitly seeks replication studies (http://www.economics-ejournal.org/special-areas/replications-1) and I like the idea of the journal. They also have some unique features embedded in their review process. But the details seem daunting. They require that you send the manuscript to the original author and give them 60 days to respond. You need to provide all the correspondence with the original authors as well. There are a number of steps which feel like they extend the process and invite the all-to-common tendency of authors to change their stories to fit the new findings. While, I do think corresponding with the original authors is good practice, it feels like it gives a sort of “veto-power” to the original study authors. The reason I am thinking this is because my study is not a traditional replication. I am using a different data source and different time period than the original study and I am not really attempting to do these same analysis they did (for which they have not released the data – it is proprietary). So, in this case, I intend to share my work with the authors and ask for their comments, but I really don’t want to feel compelled to wait for, and incorporate, their response. I’d be interested to hear what others think about the list of requirements for submission to that journal.

    Regarding the idea of legislation: I think a far simpler and more effective idea is for regulators and legislators to put researchers on notice that their work will be discounted (essentially to zero – probably epsilon > 0 in order to avoid first amendment suits) if the data and code are not provided. Much research is conducted to further both the career of the researcher and the goals of an entity that is providing the data. There is a clear desire to influence policy. But in the absence of providing the data, I see no reason why the findings should be given any weight. It does not require legislation – it only requires public servants to do their jobs (well, I suppose, that may be a long shot to hope for).

    • Keith O'Rourke says:

      > no reason why the findings should be given any weight. It does not require legislation – it only requires public servants to do their jobs

      That often is the default in some regulatory agencies.

      I would prefer random audits similar to tax audits and one university (University of Toronto) has started to do that, though at this point only to ensure ethical standards are being followed, which includes proper data management.

    • Bob Reed says:

      as Co-Editor of the Replications Section at Economics E-Journal, let me respond to Dale’s comments. We do give the original author(s) a chance to respond to a replication of their work. However, the original authors do not have veto power. Their comments are published alongside the replication. And while this adds some time on the front end of the review process, the nature of the review/publication process at Economics E-Journal, and the open access dimension of the online publishing, shortens the time to “in print” at the back end of the process. That being said, we do require that all replications make a good faith effort to reproduce the original study. This is to ensure that the replication study understands the empirical work that underlies the original study.

  2. Anoneuoid says:

    replicating experimental results can be onerous

    There are people out there who actually get pleasure from replicating research and having others try to replicate their own research (I mean no matter how it turns out). Those people actually like science.

    I know it sounds mean, but anyone who really thinks this should quit trying to do science. It means you do not actually like science. That is fine, it doesn’t make you evil or bad. It’s just like how some people enjoy exercising or fancy dinners or whatever while others don’t. Leave the science to people who enjoy doing it.

  3. Joseph says:

    I am happy to share code.

    But I do a lot of duplication due to data harmonization efforts and I think there is a bit of a mythology here that duplication (of any merit) on real data projects is easy. Just rerunning the code on the final data set will duplicate any errors in the analysis and assembly of the data, so one needs to rip the code apart. I am always astounded by the ways in which this process can go wrong, even with the code. there can be real effort involved in helping an outside researcher puzzle out how to make the code work on the data set (even setting the libraries correctly has been a barrier in my experience). I have taught classes where I give the students the code and the data, and it is amazing how many ways this can go wrong.

    I think there are also a couple of other thought-lets here. Journals sometimes want code now, but sometimes they provide no venue to provide it, especially if it is long.

    The modeling code, itself, is trivial compared to the (often huge) data cleaning and assembly code. I am on the verge of trying to find a way to sneak the modeling statements into the methods sections of papers because it would be more precise. But this longer and messier code often includes a lot of exploration and checking — I often have data checking code that runs into the thousands of lines.

    The second is a consensus on what counts as an error. Almost no code is ever completely error free. But not all coding errors impact the statistical model, the data elements in the final model, or the conclusions. How does one handle a botched transformation of a covariate that is never used in the final model? Or a model that you would never use in practice but you were curious just how well a linear regression would have worked to estimate a parameter? Or a complete case analysis in an inappropriate place to see just how much the estimate changes? Or do we want sanitized code that doesn’t look at all like real working code and strips out all of these pieces? And I have yet to see ANY non-trivial data set in the public domain that doesn’t require at least some processing — even just exclusion criteria or the accounting for missing data elements requires some thought.

  4. a reader says:

    I think legislation is silly: do we really need to raise taxes for a government agency that checks that researchers provide code? Would we have any faith in this agency?

    On the other hand, that there are journals that don’t make code a requirement for publication (outside the case of requiring access to sensitive data) in 2018 is bananas.

  5. Interesting. There are indeed far too many people taking money for “doing science” who are actually just “doing social” or “doing career”. These people should be dealt with, particularly when the public is paying them. However…

    I won’t be letting anyone see my source code. When I put forward a theory, or test competing theories (if it’s that kind of experiment), I am offering an idea and probably a Method. In an old fashioned type of science one offers the technique in enough detail for others to replicate the experiment. One does not need to offer all one’s test-tubes, Bunsen burners etc. No one at CERNE feels oblged to let just anyone have a go on the Large Hadron Collider in order to have their theories accepted (or even paid for!).

    I admit I’m not familar with the type of science where one needs multi-thousand line programs just to get the data ready to be presented to the statistical test process (or whatever the Bayesian equivalent is!). When Nathan Myhrvold re-runs dinosaur stats he just uses the data, not the experimenters software – but maybe that’s not an apt comparisons – and I see you’ve discussed NM before! https://andrewgelman.com/2016/05/23/albedo-boy-is-back/

    In my case the field should have written their own software years ago and if they have to write their own now then it’s about time. Also I’ve not accepted funding for my research so I feel entitled to sell the progam, which I couldn’t if the code was given away free. It is up to others to refute an idea, not to find all sorts of reasons to ignore an interesting new idea; demanding code can be seen as one more example of this.

    Then there is the issue touched on by Joseph, that some bugs are not very relevant. However, any tiny bug will be pounced on as an excuse to zeroise the entire research project, when so many critics are as morally defunct as they are in some subjects.

    So, no, my sourcecode won’t be released, and I don’t accept not being given the code as an excuse for ignoring the work. Of course when the software has been paid for by a grant it’s a different thing.

    • Slimy mold says:

      I don’t think the analogue with sharing test tubes or letting people run analyses at CERN works at all. I don’t think sharing analysis scripts is ANYTHING like what you are comparing it to; actually, quite the contrary: in both of these cases the direct analogue–well, I don’t know about CERN, but–is NOT something as ridiculous as what you are suggesting but–sharing the analysis scripts. My point is that you don’t need to retort to such comparisons in this case since the idea presented here is directly applicable to the cases you brought up (again, maybe not in the case of CERN).

      Also, I want to emphasize the word “script”. You are continuously using the word “software”, and maybe that’s technically correct–it’s something that’s not hardware–but the connotations are something completely different and, I feel, can cause confusion: software brings to mind something like Linux Ubuntu, Duke Nukem 3D or SPSS. Analysis scripts in most cases, or in the cases usually referred to in this blog, are something completely different: we are talking about e.g. R files containing code for reading the data, preprocessing it, and running the analysis–and maybe creating the plots.

      That being said, of course sharing scripts isn’t necessarily as straightforward as that in all cases, and people here have brought up good points. Also, I agree with “a reader”: tackling this issue through legislation seems… ugh.

  6. Hi Slimy – Yes in many cases it’s the best and obvious thing to do, but there are some cases where insisting that the code is a sine qua non, is not right.

Leave a Reply