Dominik Papies points me to this article, “Matched-Names Analysis Reveals No Evidence of Name-Meaning Effects,” by psychologist and data detective Uri Simonsohn, in collaboration with Raphael Silberzahn and Eric Luis Uhlmann, the two authors of an earlier study that this new report is refuting.
This seems to me an interesting case where a piece of research is criticized and the critics and the original authors resolve the issue together in a follow-up article. This may fit into you thoughts and writing on how to deal with method disputes.
I do think this article is notable, and Silberzahn and Uhlmann are admirable in accepting that they might have made a mistake in their earlier publication, an attitude that is disturbingly rare, not just in academic psychology but in journalism as well, as we’ve discussed often enough in this space during the past few years.
That said, I want to hold on to the hard line that, while this sort of collaboration is excellent when doing a criticism, reanalysis, or replication, and I think this sort of collaboration should be encouraged, I don’t think it should be considered as a requirement or even an expectation in such interactions. (Sorry for all the italics; I think I’ve been malignly influenced by too much reading of Dan Kahan.)
It should be clear to all (I assume) why the collaboration exemplified by Silberzahn, Simonsohn, and Uhlmann is a good thing. The reason I still don’t think such collaboration should be an expectation or a requirement is, simply, cost. I think that criticism is important enough that it should not be burdened with a necessary link to collaboration.
One way to see this is to consider the pre-publication review process. Referees and journal editors are allowed—indeed, expected—to offer methodological and substantive criticisms without the expectation that they contact the authors, visit the authors’ labs, etc. I think the same should hold for post-publication criticism. When a collaboration can be done, as in the example above (or in a recent example of my own, where a prominent survey researcher offered a bunch of helpful comments on a forthcoming paper of ours, fortunately giving us the comments in time for us to alter the paper before publication), that’s great, and it should happen more often, and it should be encouraged. Just not required.