Yup, another social psychology researcher from northwestern Europe who got results that people just don’t believe.
I’m a fan of Retraction Watch but not a regular reader so I actually heard about this one indirectly, via this email from Baruch Eitam which contained the above link and the following note:
Of the latest troubles in Social psychology you probably heard. Now, about the others I wasn’t surprised, I “grew up” in this climate. I know these practices and in a real sense the manner my lab works is molded as a negative to these practices (mostly, shared scripts and datafiles and replications, replications, replications). I personally wasted ~1.5 years of work and 600(!) participants trying to replicate one of Bargh’s experiments (at Columbia btw). Jens’ case suprised me a great deal. I met the guy and he seemed to me the most passionate and honest person. Also not one of these hotshots or ideologists that treat the data as ornamentation to their great ideas. Orthogonally to my prior I am also disgusted by the manner of the discussion. I find that the whistleblowers have turned into a bunch of bloodthirsty hunters that fail to show even a shred of doubt when at stake (with all due respect to science) is a persons’ life’s work, integrity and livelihood. I know you pitched in the discussion and we had an excellent lab around you recent paper on the non-fishing version of “researchers degrees of freedom”. I strongly resonate with this rigor squeeze and have intuitively applied many of these procedures, without much guidance, as early as my second year of doctoral studies. Still to what degree of certainty can we be sure about the conclusions based on the (weird, i must admit) linear pattern found in Forster’s data? Can we quantify the certainty of the conclusion itself? Shouldn’t at least two different statistical approaches be applied when the future of a person is at stake here? I also asked a collaborator of mine Zoltan Dienes to look into the “statistical accusations” but I would love to hear your opinion and would be more than glad if you address this on your blog as i feel it is an ethical matter at least as it is a professional one (giving the dude a fair hearing that is).
I have tried to voice this opinion on retraction watch but was attacked by a bunch of angry men and lost interest (I am from Israel, there are more serious battles to fight here…)
My reply: Figure 3 on page 9 of this report looks pretty convincing no?
The only weird thing is, why would someone make such data so linear on purpose? It makes me suspect that perhaps what Forster reported as raw data were not raw data but rather were derived quantities based on a fitted model (perhaps with the rare departures from perfect linearity arising from roundoff issues). I could imagine this happening in a collaborative project where data and analyses are passed around. Forster wrote that he “never manipulated data” but later he said “if data manipulation took place, something that cannot even be decided on the basis of the available data, it cannot be said who did it and how it was done” which makes me wonder whether there were some holes in the collaboration.
At this point, one possible scenario for Forster seems to be that something is very wrong in these papers, he knows it, and he’s trying to give denials that are misleading but literally true. I don’t know for sure, but it does seem like (a) the data are fishy and (b) he must know that these aren’t the raw data. Another possibility is that, if the raw data ever were recovered, the ultimate research conclusions might not change much. I can’t really tell, having not looked at the original papers and indeed having only skimmed the report from the university committee.
It’s hard for me to picture someone going to the trouble of making up data to look exactly linear, but I could easily see a research team, through a mix of sloppiness and misunderstanding of statistics, taking the estimates from a linear model and then plugging them in a later analysis. Sometimes this sort of post-processing makes sense in an analysis, other times it’s not the right thing to do but people do it without realizing the problem.