Rob “Lasso” Tibshirani writes:
We all read a lot of papers and often have useful things to say about them, but there is no systematic way to do this lots of journals have commenting systems, but they’re clunky, and, most importantly, they’re scattered across thousands of sites. Journals don’t encourage critical comments from readers, and letters to the editor are difficult to publish and given too little space. If we’re ever going to develop a culture of commenting on the literature, we need to have a simple and centralized way of doing it.
Last year, I [Tibshirani] approached my Stanford colleague Pat Brown, a founder of PLOS, with the idea of creating a site where scientists could comment on ANY published research article something like comments on movies at Internet Movie Data Base (IMDB) or comments on books and other products at Amazon. Pat said that he been discussing similar ideas with his PLOS cofounder Michael Eisen, and that they felt strongly that a standalone site would be unlikely to work because it would not get enough traffic. They felt that the best way to develop a successful culture of commenting on science papers would be to make this an option at PubMed. Pat introduced me to David Lipman, the Director of the NCBI (the home of PubMed), who said that the idea has been raised many times in the past, and that he was open to implementing such a system if I could demonstrate broad support in the community.
So I organized a group of 34 team leaders, representing diverse scientific fields. They recruited teams of prominent researchers in their fields 250 in all, who were committed to the idea. David took the idea to the NIH leadership, who approved the development of a pilot commenting system called PubMed Commons. The team of scientists I assembled agreed to beta test the system during development and to provide feedback on its design and operation.
Who should be able to post comments?
A central issue for PubMed Commons was the question of who should be able to post comments. One would like the system to be inclusive as possible but many scientists would not be interested in posting comments in a system with a high proportion of irrelevant or uninformed
￼comments. NIH also needed a rule for who could post that would be pretty clear cut and not based on e.g. some judgment of the experience or knowledge of the participants. The decision was made that comments could only be posted by authors of papers in PubMed. This would make the situation symmetric in that all people who comment can have their own work commented on. It would also include a large number of potential participants and would meet NIH’s need for something unambiguous. Unfortunately it would leave out many people who could add valuable input, including many graduate students, patient advocates, and science journalists. I’m a little worried about this restriction, as I want to make the system open to as many users as possible. But hopefully that is a pretty wide net, and it may be widened further in the future. And a group commenting feature to be described below could help improve inclusiveness.
Anonymous comments allowed?
One big issue that we have faced was the question of whether anonymous comments should be allowed. After much discussion, the group remained deeply split on this issue. Those wanting anonymous posts were concerned that many scientists, especially junior researchers, would be reluctant to make critical comments. But those opposed to anonymous comments believed that the quality of interchange would be higher if commenters were required to identify themselves. In the end, these differences weren’t really resolved and the decision was to start without anonymous comments and reevaluate after the system had been fully public for a while. While debating this issue various proposals were put on the table for ways to allow participants to review and essentially sponsor the anonymous post of another participant.
Gary Ward, an active member of the lead user group, was very keen on using PubMed Commons to post comments from a journal club for a class he participates in at the University of Vermont. He proposed that there should be some way for PubMed Commons to accommodate comments posted by a group. David Lipman noted that group comments would also be a way to allow participation by a wider range of commenters: A group could be initiated by a regular PubMed Commons participant (i.e. was an author of a paper indexed in PubMed), giving it a title, short description, and list of participants and then posting comments on their behalf. While a group comment could be submitted by a particular group member, in many cases, they would reflect the consensus of the group and such collective comments could be quite valuable.
PubMed Commons is here!
The NCBI team developed a working version of PubMed Commons earlier this summer and I posted the first comment in the closed pilot on June 17. Since then the user group has noted bugs and made a number of requests for modifications. Jonathan Dugan of PLOS labs pulled together members of the publishing world for strategic advice, and has provided many valuable
￼suggestions about the design of the system. The current system is pretty simple after registering you’ll see the PubMed Commons landing page which has all the most recent comments and links for information on how to use the system. When you’re signed in you’ll see below each PubMed record a box for posting comments or replies to existing comments as well as a place to indicate that an existing comment or reply was useful. There are instructions for how to specify simply formatting of a comment and if you cite another PubMed record in your comment, there are links back from that cited paper to your comment.
We believe the system is now ready for a wider range of participants. If you’ve been funded by an NIH Extramural grant (or in the NIH Intramural program), NIH has the information it needs to get you into PubMed Commons automatically. Once you’re a registered participant, you can invite other published scientists to join. NCBI is investigating ways to open Commons up directly and automatically to more groups of published scientists but if new participants invite their colleagues, the network effect could broaden membership and expand participation dramatically.
The system will still be in a closed pilot mode where only registered participants can see the posted comments but NIH leadership will be evaluating the closed pilot with the hope of making all comments visible to all users of PubMed. All comments are covered by Creative Commons Attribution license (http://creativecommons.org/licenses/by/2.0/ ) and if the decision is to make the system fully public, NCBI will provide an API so that other groups (e.g. publishers or other information resources) can make these comments useful to the community.
I’m not really part of the PubMed world so I can’t comment on the specifics, but from here it looks like an excellent idea.
P.S. Further discussion here from Ivan Oransky who, like some of our commenters here, is unhappy that the new system will restrict the number of people who can comment. I understand this concern, but to me it still seems like a step forward to have commenting that is “official.” I worry that, now, editors and readers of journals such as Psychological Science can simply dismiss commenters as coming from bloggers etc., whereas if the comments are directly attached to the article, maybe they would be taken more seriously. I wouldn’t mind if my own articles had comments from others attached—although I guess I’d be upset if we ended up with the kind of thing we see in the comments sections of newspaper blogs.
In this blog, we’ve had excellent experiences with completely open commenting, almost no trolls at all, and often even the angry commenters have something useful to say. I don’t know how we’ve managed to be so lucky. But I suppose having “statistical modeling” in the blog title is a filter that keeps out the truly thoughtless people. If PubMed had a completely open comment system, I’d guess that most of the time things would go just fine, but maybe there’d be problems with hot-button issues such as vaccines (is that the medical equivalent of a blog post on the Israel-Palestine issue?) and maybe problems with sock puppets on any articles that are related to big-money drugs.