Let’s be open about the evidence for the benefits of open science

A reader who wishes to remain anonymous writes:

I would be curious to hear your thoughts on is motivated reasoning among open science advocates. In particular, I’ve noticed that papers arguing for open practices have seriously bad/nonexistent causal identification strategies.
Examples:

Kidwell et al. 2017, Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency. Published in Plos Bio, criticized in Plos blog here. Brian Nosek responds at great length therein.

McKiernan et al. 2017, Point of View: How open science helps researchers succeed, claims that there is “evidence that publishing openly is associated with higher citation rates,” but also notes that “some controlled studies have failed to find” an effect. A glance through the citations suggests that all the RCTs find null effects, and the observational studies find substantial effects.
3) Rowhani-Farid and Barnett 2018, “Badges for sharing data and code at Biostatistics: an observational study” — compares citations for articles in a journal (biostatistics) that introduced badges to those in a journal that did not (statistics in medicine); the causal identification strategy is that the two journals are “in the same field of research with similar goals of publishing papers on statistical methods development in health and medicine.” The article finds that the “effect of badges at Biostatistics was a 7.3% increase in the data sharing rate.” (Further down, they write that their study “cannot accurately deduce the effectiveness of badges because of the biases of the non-randomised study design.” Well, then how should we interpret the claims in the abstract??)
So, one thing that makes me feel sad about this is that they’re all published in journals with a clear stake in open access norms — PLOS, eLife, and F1000. I worry that publishing articles like these discredits the model.
Also, I do think there are large, well-justified benefits to open science practices. Davis 2011 finds that “[a]rticles placed in the open access condition (n=712) received significantly more downloads and reached a broader audience within the first year, yet were cited no more frequently, nor earlier, than subscription-access control articles (n=2533) within 3 yr.” David Donoho (2017) writes that “[w]orking from the beginning with a plan for sharing code and data leads to higher quality work, and ensures that authors can access their own former work, and those of their co-authors, students and postdocs.” But I guess that there is still demand for research showing a strong citation benefit to open scholarship, regardless of what the evidence says.

I don’t have the energy to read these papers—I guess I don’t really care so much if open-science increases citation rates by 17% or whatever—but I agree with the general principle expressed by our correspondent that it’s not good practice to exaggerate evidence, even in a good cause.

25 thoughts on “Let’s be open about the evidence for the benefits of open science

  1. Sometimes you do things based on principle, not evidence.

    In this case it seems to be that the evidence people are looking for is a personal benefit to the author, higher citation rates. Open access isn’t about that. It’s the right thing to do in principle and there is no evidence that it drops citation rates. That seems sufficient to me.

    • There’s also something to be said for “open science” being poorly-defined. It’s been said that the idea of reproducibility is not yet, by itself, reproducible. Until that definition can be nailed down, it’s tough to empirically evaluate whether or not it’s working.

      Nevertheless, I completely agree with your distinction between evidence and principle, and I also happen to wholeheartedly agree with the principle. I also think there are a lot of potential empirical benefits to long-term, multidisciplinary, well-defined practice of open science. But just the fact that it’s the right thing to do on principle is sufficient for me as well; if any potential empirical benefits play out down the road, even better.

    • The principle though isn’t just “This is the right thing to do” in some abstract sense (like say “god wants me to” or similar reasons) the principle is based on utility of the scientific output and efficient use of economic resources. As a society we don’t want to give money to people who pretend to study things and then tell us stuff that is wrong and/or causes harm. We want to find out something at least *approximating* truth about the world. The reason to use open science practices, particularly *open data* is that it allows one to check the results of the study and to use the data in combination with other data to answer other questions. It dramatically improves the economic value of the work.

      That might even be *especially* true if it *harms* citation rates and leads to people *washing out* of science, specifically people who don’t know how to do good science and are all set to make a name for themselves on poorly thought out uncheckable Wansinking.

      • Some commentary I’ve come across on Twitter does sound as if some % of researchers dwells on what is wrong with research. That same % generally gets into high-fiving colleagues with whome they agree. from my observation. To an extent I understand that it is based on sense of ‘solidarity’ behind a particular approach to research. But that same % should also contribute to how to ‘advance’ the research flaws. Or be upfront that it is still in the ‘exploratory’ stage.

        Then too funds are limited. At different points, within last couple of years, some have highlighted this reality. However subsets of them endorse still statistical practices that unequivocally entail weak evidence. In other words, they endorse, transparency, data share, replicabillity, peer review. But they are not able to forge a credible articulation of theory. This is one reason I think Paul Rozin has nailed the problem in at least social psychology.

        In short, we spend far too much effort on quantification to begin with. That’s why blogs, preprints, registration are good avenues to express our viewpoints.

        • There are, I think, a large number of people who would like to be in academia but feel that what it takes to actually succeed is too much bullshit and not enough actual learning about the world. Let’s suppose half of good scientists are in this boat. The other half put up with the academic stuff for various reasons…. But still this doesn’t make up all the academy, no I’d estimate that something like 50 to 70% of academics are in some sense “clock punching”. Not really furthering the goal of knowledge, either due to low competence, focused on a mostly meaningless field (high performance computing applied to climate prediction in 100 years, or quantifying the fractal spectrum of certain matrices arising out of pure math abstractions of observed ant foraging behavior, or whatever), or worse yet active incompetence (Wansinking etc). My guess is if you’re one of these people, you probably don’t read Andrew’s blog regularly, so the group we get here is mostly the good and motivated researchers.

          Nevertheless, if we can dichotomize in some sense in the way I just did, we could, with sufficient knowledge, cut the academic workforce by say 70% and then hire back all the people forced out of academia due to being too careful and too interested in real-world correct research findings, and therefore not able to produce the kind of stream of three-times-annual grant-fundable “dramatic, significant new findings” baloney needed to keep your operation funded.

          Of course, the same could be said for the US Congress, and I don’t hold any hope that it’ll wind up full of good and responsible caring civic-minded people full of desire to improve the lives of everyday americans any time soon either.

        • You wrote: “no I’d estimate that something like 50 to 70% of academics are in some sense “clock punching”

          I have some (perhaps amusing) anecdotal data regarding your estimate:

          Around 2011/2012 (just before the “reproducibility crisis” hit) i, a recently graduated research master student, learned about all the problematic issues in psychological science via twitter/blogs/etc. This was just before the special “replication issue” of Perspectives on Psychological Science was published (http://journals.sagepub.com/toc/ppsa/7/6). I never heard about any of these problematic issues during my education, which was not only terribly shocking to me, but it felt unethical and irresponsible.

          But what could i do about it, i had graduated and left that place…

          With the sole intention of trying to help improve matters concerning 1) psychological science in general, and 2) the education for future students at my former university, i (perhaps foolishly) e-mailed all my professors, and the head of the university, in a group mail to point them to an online article about the possible benefits of student replications, and the importance of replications in general.

          I also used the subsequent back-and-forth discussion with a few people to point them towards all the (crucial) things that i was not (really) taught (e.g. publication bias, replications, p-hacking, etc.)

          Anyway, out of 33 or something like that (in many cases my own former) professors/teachers, 4 replied, of which 3 seemed to at least acknowledge the possible importance of the issues i mentioned and felt were left out of my education.

          So, i interpreted this whole thing as my “anecdotal data” showing that roughly 90% of professors might just be “punching the clock” and may have totally forgotten why they started with science in the first place…

          Or they could have just thought i/it was just all too weird…

        • Daniel,

          There are far too many researchers for sure. I continue to hold that credentialing is not necessarily the means by which science has improved. It’s fluid & crystallized intelligence that that underpin it. And my sense is that pedagogies in some fields dampen both. Particularly in the biomedical fields as a consequence of medical education. I would include social sciences too. This is why I am keen on the sociology of expertise as one facet of knowledge formation and acquisition. It’s of course not the only facet. Attribution in decision making has played a disproportionate role. Therefore it has prevailed in ways that waste time and energies.

    • “Sometimes you do things based on principle, not evidence. ”

      I agree.

      I just want to note that the anonymous person in the blog post (a different “anonymous” than the one who is writing this comment) may also totally agree with this.

      In my reasoning, it may still hold that “(…)motivated reasoning among open science advocates” is important to keep in mind and that “In particular, I’ve noticed that papers arguing for open practices have seriously bad/nonexistent causal identification strategies”.

      In other words, regardless of whether evidence matters in any of this, *if* you are going to gather/present evidence, it seems important to keep possible motivated reasoning into account, and be critical concerning the possible evidence.

      In light of this, i find the title of Kidwell et al.’s 2017) paper sub-optimal because (if i am not mistaken) only 2 out of 3 used badges at the time seem to have been examined in the paper: open materials & open data (and the pre-registration badge seems to have not been). Furthermore, the open materials badge seems to have had a less clear beneficial effect compared to the open-data badge. Together, at least to me, the title is therefore sub-optimal at best and misleading at worst.

      (Side note: in the paper it is stated that “B. A. Nosek created the badges to acknowledge open practices (…)” and in the PLOS one blog Jon Peirce stated that “(…) I’m also on the Badges committee, indeed I originally proposed their creation (…)”. So, perhaps we have a case of “multiple discoveries” (https://en.wikipedia.org/wiki/List_of_multiple_discoveries) here!)

  2. From the Kidwell et al. (2017) paper: “In this article, we report evidence that a very simple incentive—badges acknowledging open practices—can dramatically increase sharing of data and materials.”

    It is still unclear to me how and why badges awarded by journals on their papers are an “incentive”. Can anybody explain this to me…

    (Side note: i have written on this blog before that i reason using words like “incentives” might be sub-optimal because i reason it is an indirect general term for other things that in turn 1) might stop you from really thinking about things, and 2) could (therefore) lead to coming up with a whole new set of “incentives” that may not even be good for science).

    • I think people are reasonably likely to see the badges as a shorthand for “good article” and authors are likely to act accordingly to get these accoutrements if that is the case.

      • I looked up the word “incentive” and it can be defined as “something that encourages a person to do something” (https://dictionary.cambridge.org/dictionary/english/incentive)

        You wrote: “I think people are reasonably likely to see the badges as a shorthand for “good article” and authors are likely to act accordingly to get these accoutrements if that is the case.”

        Does this mean that badges are really an indirect “incentive”, the real “incentive” being that others view your paper as being a “good article”?

        If so, i wonder if there is a possible danger in this possible indirect way of going about things. I wonder if the more you speak in general terms about “incentives”, and the more indirect you let them be (-come), the higher the chance could be that you end up with a new set of “incentives” that may not even be good for science.

        • I wrote:

          “If so, i wonder if there is a possible danger in this possible indirect way of going about things. I wonder if the more you speak in general terms about “incentives”, and the more indirect you let them be (-come), the higher the chance could be that you end up with a new set of “incentives” that may not even be good for science.”

          To (possibly) illustrate my point using 2 examples:

          1) Here is, what i believe is, the editor-in-chief of the journal Psychological Science. If i am not mistaken, this is the same journal who implemented the “open practices badges”, and was part of the Kidwell et al. (2017) study/paper mentioned above:

          http://statmodeling.stat.columbia.edu/2018/04/15/fixing-reproducibility-crisis-openness-increasing-sample-size-preregistration-not-enuf/#comment-712159

          In this discussion thread the editor-in-chief of the journal Psychological Science wrote that:

          “I don’t know how many submissions in 2017 that included one or more preregistered studies were declined, but I do know that at least some were.”.

          To me, this is a (possible) sign of (possibly) losing track of why and how we should be improving science. If the goal is to “improve science”:

          * Shouldn’t the editor-in-chief (want to) know how many submissions included eligible-for-badges practices?
          * Shouldn’t there (also) be a study that investigates the actual amount of papers performed and/or submitted that used eligible-for-badges-practices? (not just the ones that got published)
          * Are we all possibly missing the point if we all start equating badges handed out by journals on papers with them being a “good article”?
          * And, perhaps even more importantly, are we giving journals and editors (i.c. the system that probably largely messed things up in the 1st place) unnecessary “power” to possibly influence matters by making a “big deal” out of badges on papers they themselves hand out?

          Perhaps editors can “pick out” their favorite papers/authors/topics/results who are eligible to be rewarded with badges and only publish those in their journals. The possible non-favorite papers/authors/topics/results (who could just as well be eligible to be rewarded with badges) could simply be discarded, and/or only non-favorite papers/authors/topics/results WITHOUT eligible-for-badges-practices are published. This in turn could result in a whole new way of creating “publication bias” and/or misrepresenting the totality of all performed studies, but not based on significance but something else…

          2) In my reasoning you should give the “power” and responsibility for (good) science to the scientists and the reader, not to the journals/editors/system who (probably) messed things up in science in the 1st place.

          In light of this, i thought it could be a possibly useful idea to give scientists the possibility to use “open practices badges” on their CV’s (if they want to). To me, this makes much more sense compared to using badges on papers handed out by journals, because i reason the “incentive problem” narrative of the past few years seems to me to largely revolve around the issue of scientists being rewarded for “publishing lots of papers” and evaluation committees “only looking at the no. of published papers and in which journal they appeared”. To me, that’s the most important place where you want to provide an alternative and “disrupt” the problematic process.

          Should you still have the energy left to read some more, the following discussion i (“a” on that forum) had on the Open Science Framework discussion board regarding the use of badges on CV’s was, and still is, incomprehensible and worrying to me:

          https://groups.google.com/forum/#!topic/openscienceframework/7oeul9pusKg

          I am getting more and more worried that a lot of these “improvements” and “nudging of the incentives” could be giving even more “power” and “influence” to the exact system and people who (probably largely) messed things up in the 1st place…

          Next thing you know, they will want to create special “open practices badges experts” at the journals who will “check” whether everything is okay. Or, they will make fancy electronic badges that could in turn only be issued by journals. Etc. Etc.

  3. tl;dr but one thing to be careful about in observational studies of this sort is that most high ranked journals are still closed access. That is where most top authors publish, and where they send their best papers, so you would expect more citations.

    Moreover, the focus on citations is also a little myopic. Other measures like downloads, almetrics etc may also be relevant. Indeed, the popularity of Sci-hub being a case in point. In addition, likely there are some papers that would have never seen the light of day, were it not for the more enlightened policies of OA journals. And so on.

  4. Wow what a timely blog post b/c I was reading an article on a related topic that an epidemiologist sent me.

    As Daniele Fanelli pointed out in a keynote, it is an open question whether we can bill the current state of research as a crisis. But in the 90’s, I was then reading articles on the Evidence Based Movement. It was deemed as a crisis. In some part the HIV/AIDS situation contributed to the exigency for finding the cause/es and treatments. I gather that effort yielded some treatment successes. Today we are debating whether there is a crisis. So that’s a bit puzzling.

    Fernando is right that the focus on citations is little myopic. I particularly like the point that many more papers that would have not bee accepted by prominent journals are available.

    There are only a few in any given scientific circle that have superior diagnostic/prognostic talents, whether by way credentialing, practical experience, fluid intelligence, and crystalized intelligence. We should encourage, therefore, more latitude in research efforts. Transparency is an essential norm toward improving science. Such talented individuals are not so easily led around by their career interests. Some larger % of them has a moral compass by virtue of their backgrounds, from what I have gleaned.

    I think that John Ioannidis and Brian Nosek have demonstrated that they are endorsing more discussion and debate, by way of actual collaborations and education. So that is heartening.

  5. The number of people who want to cite my work, but don’t have access to the journal, cannot get it via ILL, can’t find the free version on my website, and can’t be bothered to email me for a copy, is probably pretty close to zero.

    It’s pretty telling that nobody has done a welfare analysis for switching to OA — and I think it’s because it just wouldn’t work out favorably. Harvard Library complained after a recent price hike that it couldn’t afford the $3.5m/year subscription fees. They have about 2,400 faculty. If each of them published two papers a year and paid a $1,000 publication fee, that’d be $4.8m/year in publication fees. But the average faculty likely publishes more than two papers and the average publication fee is about $4,000. So how much money would they really save?

    Never mind all the switching costs associated with such a move: it’s pretty difficult to abandon the top journals and establish OA versions that somehow have the same reputation. We see how well that’s going with Science Advances, for example: some good papers, but clearly not equivalent to Science. And that’s with all the expertise, infrastructure, and a brand name to start with — not starting something from scratch.

    But suppose that we could get publication costs down to an impossible zero. You’ve now saved Harvard $3.5m out of their $4.5bn operating budget. That’s less than 0.1%. Sure, not every university is Harvard — but other universities also pay lower subscription fees. I find it pretty unlikely that any university budget is remotely affected by subscription fees. If you want to cut costs in education, that’s not the place to look.

    • Great points. It’s obvious that OA just shifts costs from institutions to authors/funders, but the welfare analysis is an interesting thought, and if it’s even close to correct, it’s another argument against “pay to publish” OA.

    • The number of people who want to cite my work, but don’t have access to the journal, cannot get it via ILL, can’t find the free version on my website, and can’t be bothered to email me for a copy, is probably pretty close to zero.

      Maybe I am strange, but usually when I “read” a paper it is more just skimming it since it popped up during a search looking for some nugget of information or to see what the media is trying to tell me. In neither case would I work harder than searching for the authors webpage.

      But it sounds like you only care if people who are likely to cite your paper read it? Really? Also, is your website going to be around in 10-20 years?

      • “But it sounds like you only care if people who are likely to cite your paper read it? Really?”

        The psychological damage done by being in Academia is not to be discounted. I had an opportunity to discuss with some people the possibility to collaborate on a project involving studying healing of certain wounds. When asked whether they wanted to consider participating one of the PIs said “Well, since we will be able to publish in journals like Science and Nature and there are no competitors in this subfield, I’d say we are all for it!”

        No mention was made of whether answering the question of how wounds heal spontaneously was of any interest to them.

        • Sad. Makes me wonder if our educational system has focused too much on giving out stars and stickers more than on doing things because they are worth doing?

        • Rephrasing Adam Smith: It is not from the benevolence of the PI, the professor, or the PhD that we expect our scientific progress, but from their desire to brag about their Nature paper during the conference dinners.

        • It is not from the benevolence of the car salesman it’s from the desire to brag to his coworkers about how many previously totaled salvaged lemons he sells with falsified title by lying?

          Sometimes incentives don’t produce actual quality product, in that case, the incentive system doesn’t work. This is increasingly the case across vast swaths of the economy, not just science. The words “mortgage crisis” come to mind.

        • +1
          I dont think that Adam Smith was right in the first place, but it is still fascinating how some institution continue to function more/less efficiently even after their inner motivation is completely corrupted.

Leave a Reply to psyoskeptic Cancel reply

Your email address will not be published. Required fields are marked *