Evan Heit sent in this article with Caren Rotello and Chad Dubé:
There is a replication crisis in science, to which psychological research has not been immune: Many effects have proven uncomfortably difficult to reproduce. Although the reliability of data is a serious concern, we argue that there is a deeper and more insidious problem in the field: the persistent and dramatic misinterpretation of empirical results that replicate easily and consistently. Using a series of four highly studied “textbook” examples from different research domains (eyewitness memory, deductive reasoning, social psychology, and child welfare), we show how simple unrecognized incompatibilities among dependent measures, analysis tools, and the properties of data can lead to fundamental interpretive errors. These errors, which are not reduced by additional data collection, may lead to misguided research efforts and policy recommendations. We conclude with a set of recommended strategies and research tools to reduce the probability of these persistent and largely unrecognized errors. The use of receiver operating characteristic (ROC) curves is highlighted as one such recommendation.
I haven’t had a chance to look at this but it seems like it could be relevant to some of our discussion. Just speaking generally, I like their focus on measurement.
The starting sentence was an interesting euphemism. No, Psych Research has not been immune to the crisis; but some might say it is the epic-center of the crisis.
Thanks Andrew.
Rotello, C. M., Heit, E., & Dube, C. (in press). When more data steer us wrong: Replications with the wrong dependent measure perpetuate erroneous conclusions. Psychonomic Bulletin & Review.
http://faculty.ucmerced.edu/sites/default/files/eheit/files/rotello_heit_dube_pbr.pdf
Great paper! I think the main idea applies to other domains with non-normal nonlinear measurements in psychology. For instance analyses of choice responses and reaction times often ignore the possibility of speed-accuracy trade-off which can lead to misinterpretation.
I have two small quibbles. 1. It is straightforward to extend signal detection theory to obtain asymetric ROCs by introducing additional parameters. This will create a model that is strictly not identifiable (3+ pars but only 2 data points). However, this can be resolved with the help of hierarchical modeling – by pooling parameters across subjects.
2. In your second point on page 10 you propose to use point estimates of ROCs obtained by previous studies. The point estimates however neglect that these studies report only estimates. In addition, any following analysis should take the uncertainty of these estimates into account. Probably, the best way to do this would be to estimate a single model that uses old data alongside any new data.
Evan: Folks may have been more aware of this kind of issue in Epidemiology.
For example – Spurious precision? Meta-analysis of observational studies. http://www.ncbi.nlm.nih.gov/pubmed/9462324
Or as discussed in Greenland S, O’Rourke K. In: Modern Epidemiology (Rothman KJ, Greenland S, Lash TL, eds). 3rd ed. Philadelphia:Lippincott Williams, 652–682; 2008. Meta-analysis.
Hello, it seems that the second link is wrong (404)
This blog is fascinating, thanks a lot.
M