Jonathan Sterne sent me this opinion piece by Stephan Lewandowsky and Dorothy Bishop, two psychology researchers who express concern that the movement for science and data transparency has been abused.
It would be easy for me to dismiss them and take a hard-line pro-transparency position—and I do take a hard-line pro-transparency position—but, as they remind us, there is a long history of the political process being used to disparage scientific research that might get in the way of people making money. Most notoriously, think of the cigarette industry for many years attacking research on smoking, or more recently the NFL and concussions.
Rather than saying we want transparency but we don’t want interference with research, I’d rather say we want transparency and we don’t want interference with research,
Here are some quotes from their article that I like:
Good researchers do not turn away when confronted by alternative views.
The progress of research demands transparency.
We strongly support open data, and scientists should not regard all requests for data as harassment.
The status of data availability should be enshrined in the publication record along with details about what information has been withheld and why.
Blogs and social media enable rapid correction of science by scientists.
Scientific publications should be seen as ‘living documents’, with corrigenda an accepted — if unwelcome — part of scientific progress.
Freedom-of-information (FOI) requests have revealed conflicts of interest, including undisclosed funding of scientists by corporate interests such as pharmaceutical companies and utilities.
Researchers should scrupulously disclose all sources of funding.
Journals and institutions can also publish threats of litigation, and use sunlight as a disinfectant.
Issues such as reproducibility and conflicts of interest have legitimately attracted much scrutiny and have stimulated corrective action. As a result, the field is being invigorated by initiatives such as study pre-registration and open data.
Here are some quotes I don’t like:
Research is already moving towards study ‘pre-registration’ (researchers publishing their intended method and analysis plans before starting) as a way to avoid bias, and the same strictures should apply to critics during reanalysis.
All who participate in post-publication review should identify themselves.
I disagree with the first statement. Pre-registration is fine for what it is, but I certainly don’t think it should be a requirement or stricture.
And I disagree with the second statement. If a criticism is relevant, who cares if it’s made anonymously. For example, there seem to be several erroneous calculations of test statistics and p-values in the work of social psychologist Amy Cuddy. Once someone points these out, they can be assessed independently. On the other hand, it could make sense for pre-publication review to be identified. The problem is that pre-publication reviews are secret. So if someone makes a pre-publication criticism or endorsement of a paper, it can’t be checked. It wouldn’t be bad at all for such reviewers to have to stand by their statements.
And here’s Lewandowsky and Bishop’s summary chart:
This mostly seems reasonable although I might make some small changes. For example, I don’t like the idea that it’s a red flag if Dr A promotes work that has not been peer-reviewed (a concern that is listed twice in the above table). If you do important work, by all means promote it right away, don’t wait on the often-slow peer-review process! We do our research because it’s important, and it seems completely reasonable to share it with the world.
What’s important to me is not peer review (see recent discussion) but transparency. If you have a preprint on Arxiv with all the details of your experiment, that’s great: promote your heart out and let any interested parties read the article. On the other hand, if you’re not gonna tell people what you actually did, I don’t see why we should be reading your press releases. That was my problem with that recent gay gene hype: I don’t care that this silly n=47 paper wasn’t peer reviewed; I care that there was no document anywhere saying what the researchers actually did (nor was there a release of the data).
What the researcher said regarding a critical journalist was:
I would have appreciated the chance to explain the analytical procedure in much more detail than was possible during my 10-minute talk but he didn’t give me the option.
There was time for a press release with an interview and a publicity photo, but no time for writing up the method or posting the data. That’s a flag that it’s safe to wait a bit before writing about the study.
The “peer-reviewed journal” thing is just a red herring. I completely disagree with the idea that we should raise a red flag when a researcher promotes work before publication. I for one do not want to wait on reviewers to share my work.
Also, one more thing. The above table includes this “red flag”: “Are the critics levelling personal attacks? Are criticisms from anonymous sources or ‘sock puppets’?” That’s fine, but then let’s also raise a red flag when researchers do this same behavior. Remember John Lott? His survey may have been real, but his use of sock puppets to defend his own work does give a credibility problem.
In any case, I think Lewandowsky and Bishop do a service by laying out these issues.
Full disclosure: Some of my research is funded by the pharmaceutical company Novartis.