Cristobal Young points us to this post on replication packages; he writes, “we found that only 28% of sociologists would/could provide a replication package.”
I read the comments. The topic arouses a lot of passion. Some of the commenters are pretty rude! And, yes, I’m glad to see this post, given my own frustrating experience trying to re-analyze a sociology study. To me, one of the big problems is the idea that once a paper is published, it is considered to be Truth, by the authors, by promotion committees, by the ASR and the NYT. Take that away, and everything changes: all of a sudden there’s not so much incentive to hide your data.
I don’t get the “considered to be Truth” critique. Forget promotion committees. But what about the public at large? If a layperson sees a result on the front page of a 100 year old reputable Academic Journal is he wrong to expect some “truth” in the result?
When you listen to a story on the BBC don’t you expect truth in it? The problem isn’t so much with NYT that believes the primary result but with the sociology result that prints that iffy study in the first place.
No one is saying particular articles can’t be mistaken once in a while or that the truth might be tweaked. That’s the nature of science and discovery.
But why this total aversion for the truth? Instead of sending out this nihilistic “Don’t expect the truth from us” message, why not strive to print better quality results that are more likely to be the truth?
Better curation. Quality over quantity. Shouldn’t that be the mark of a good Editor? That by whatever means, intelligence, diligence, intuition, or astrology, over time, his Journal prints a large fraction of Papers that end up being the “truth”?
Rahul:
Some truth, sure, but subject to revision. If someone publishes a result and it’s checked from all angles, we can have more trust in it. If someone publishes a result and refuses to share the data and throws mud at replication efforts, that’s a situation where the prestige of publication is getting in the way.
Andrew:
The problem in these sociology journals is “too little truth”. You can almost smell the crap. I think their editors are either asleep or in on the racket.
Is there even the desire to publish “truth”?
That’s why I don’t like the dilution of the expectation that a Journal has to publish “truth”.
I know this is too expensive to do in practice because people need to publish fast to get tenure etc, but including a replication of the novel result would be very convincing. This is hard to do in sociology probably, but not so hard in lab based psych experiments. Most work would not get published, though and many people would find themselves without a job due to too few publications.
But maybe focusing on educating the public is a better way than trying to effect these internal changes. After all, even a study with a replication should be open to question (until further evidence from other labs accumulates).
Aren’t there any better ways to decide whom to grant tenure than mindless publication counts?
? If a layperson sees a result on the front page of a 100 year old reputable Academic Journal is he wrong to expect some “truth” in the result?
Well, not if you read some of John Ioannidis’ papers or perhaps recall the Reinhart & Rogoff problem of two or three years ago, or even just read Retraction Watch.
A paper published in a reputable journal should carry some minimal quality control assurances if it has passed a decent peer review but reviewers don’t have the code or the data sets.
In some cases it may come down to a simple error, in a lab there may be accidental contamination, in a field study the sampling procedure may have broken down, in some cases the publicly available data may not be exactly what the researchers think it is. There is always a possibility of coding error.
In some cases we have outright fraud.
So no, even for laymen it’s a case of caveat emptor.
Well, that’s a great argument for giving reviewers code & data sets, no?
Oh, I’m all for it, unless you are bound by legal or other confidentiality agreements.
I disagree with this statement:
“To me, one of the big problems is the idea that once a paper is published, it is considered to be Truth, by the authors, by promotion committees, by the ASR and the NYT.”
At least most scientists I know (I’m in biology/physics, not sociology, but I suspect it’s not too different) don’t regard any published papers as “truth.” At least half of journal clubs in my lab end with everyone trashing the paper and deciding that the results are BS. I think the problem with releasing raw data/replication packages/etc, is that authors have no incentive to do so, because it can really only hurt them. A lot of those same authors probably think that it would be a good thing if other people had to make their data public, but they have no incentive to be the first ones through the gate. Authors have to be forced to release data and be transparent, either by institutional rules, or by a strong cultural norm, or more likely both.
Do I know how to do this? No. But just changing people’s attitudes about the truth/accuracy of papers isn’t the answer. The fundamental incentive structure of scientific publication is the real problem.
I was going to pos something like this before. In sociology no one every thinks an empirical article is “the truth” in my experience. It’s in the culture of the discipline to trash articles and something like the example Andrew tried to write the letter about certainly would not be seen that way. And sociology is a multi theory field so the thinking other people’s research is flawed not correct is part of the air we breathe. I don’t actually think the NY Times believes these things are True either, just that they are interesting enough to write a popular article about. I mean I guess there are people who really love their findings and will fight for them to the end, but that doesn’t mean that other people think that.
I do think that there is a disciplinary culture issue. I was working for a while on a curriculum project that included a colleague in math. Periodically we’d look at adding an article about which he would comment “but it has been criticized.” And the social science people would all say “that means people found it worth talking about.”
Is there a within-/between- issue here though? Even if the discipline as a whole takes a critical stance to empirical papers because people have an incentive to criticize the work of people with whom they disagree philosophically/ideologically/theoretically, the problem with accepting papers as True still exists within each philosophic branch of the field, right?
Maybe better said: likely few academic researchers believe a particular finding is the Truth, but people do believe that once they published “evidence” of something, that evidence should be immune from attack. And I think that is because of three reasons:
1 – some people feel like they are in a struggle of some sort against an opposition that has bad intentions against them.
2 – some researchers believe that they did a bunch of hard work and someone shouldn’t be able to come along and get a cheap publication by just changing a few lines in their statistical analysis files.
3 – some people who publish empirical papers with very strong claims actually believe they are capital-R Right, and some of those people do not like being told they are wrong because their feelings of self-worth are based on being smart and right.
I think 1) is wrong because learning should be a communal struggle against our human selves, not a professional struggle against each other; I think 2) is wrong because you are being paid to create knowledge, and sharing technology is a huge boon to knowledge production; and I think 3) is wrong because Science doesn’t and shouldn’t care about your feelings (except of course in the case of research on feelings).
And since people can’t own up to 1, 2 or 3, they go with the argument from Truth: “my paper was peer reviewed and published, so you have to accept it as True.” And people extend this “courtesy” to those they ideologically sympathize with.
Also: I’d kill for a meat pie right now. All friggin day with the cravings. Thanks for reminding me how much I miss those Andrew.
jrc:
I think you will find the following interesting: http://paulromer.net/protecting-the-norms-of-science-in-economics/
and Romer’s subsequent posts on the same topic.
Honestly, I think most people are happy when others pay attention to their work. You might see someone at the individual level defending their ideas and models to the death but even among people who are sympathetic to each other’s work you find a lot of critiquing (and that’s what we would call it really). I mean I can say that for example in one area of work I’ve done on neighborhood levels of crime that people disagree sometimes about how to operationalize concepts (like disadvantage) or they might think the way you have handled census tract border changes is not right, but I think we all know that all research is flawed.
Elin,
I agree with your points. My post was an attempt to describe how this “it was published” defense arises for some researchers, even in fields where competing ideologies promote critical engagment.
But there is an element of me missing the forest for the trees there, at least in my writing if not in my (totally unobservable) thinking. So I should provide a counter-point from my own experience.
I’ve been involved in several projects that are at least partly “replication” of some sort. In every single case, I’ve had good and helpful interactions with the researchers whose work I was re-analyzing. People who had no obvious reason to be nice to and supportive of me, other than the fact the we are both in the same profession working on the same topics.
So in the big picture, I think I mostly agree with you, and I should’ve made that more clear. My comment was about the subset of researchers who do use the “argument from publication” and how the dual-camp, ideological-divide argument doesn’t necessarily guarantee there is productive and lively critical engagement*. But like I said, I wasn’t clear on that, and my actual experience is much closer to what you describe.
*as pk notes, Paul Romer has a very interesting discussion on how the “bad equilibrium” came to dominate in an ideologically divided macroeconomics field and the consequences of that for the history/development of the field.
Upon discussing the replication issues with some of my researcher friends, (biologists) their reply has always been that a single paper does not end a discussion. Every paper is just one argument for its conclusion and the conclusion is accepted only if many papers support the conclusion from a variety of angles. From that point of view, replication isn’t quite as important. If the lack of replicability is because your theory is wrong, few other papers will successfully support it and it will fall by the way side.
I think the whole issue of replication in sociology is interesting because as a field it is in some ways late to the game. On the one hand all of the major data sets (GSS, CPS, NELS, Add Health, HSB etc) are publicly available for anyone to rerun. And data for smaller projects go to ICPSR or elsewhere pretty routinely, definitely if there is federal funding. On the other people make a million small decisions about recodes and weighting and they don’t necessarily document them well. That I think is the real issue.
You aren’t going to replicate the GSS but you should be able to reproduce the analysis.
Actually reading over you ASR story again, one thing that does really irk is the IRB issue. I guess perhaps Hamilton was using the restricted access data but in general I think there needs to be a saner approach on IRB issues for running replication analyses. REally in the present regime I don’t know that it would be allowable to download analyze individual level replication data if there was the possibility of publishing a note or doing a presentation about the results.
Andrew,
Not sure if you have seen it, but here is the latest replication attempt in Sociology. The researcher apparently had difficulty getting the data or code from authors, and then after constructing the data himself, he found the findings were wrong. It deals with an ASR article by Brooks and Manza. Sociological Science just published the replication attempt.
https://politicalsciencereplication.wordpress.com/2015/07/15/guest-post-the-replication-road-scientific-detour-or-destination-by-nate-breznau/
https://www.sociologicalscience.com/articles-v2-20-420/