Hal Pashler writes:

I [Pashler] thought you guys would enjoy this charming little 1950 paper by Edward Cureton entitled “Validity, Reliability, and Baloney” (Dirk Vorberg, a German math psych guy, sent it). Long before machine learning, it seems that psychometrics people were confronting this issue–and the concrete form it took was “What should we make of validation measures computed with the same data that were used to select out particular items for inclusion in the test?”. Just swap voxels for items, and it’s the same problem [as in the Vul, Harris, Winkelman, and Pashler paper on suspiciously high correlations in bran imaging studies].

This reminds me of a longstanding principle in statistics, which is that, whatever you do, somebody in psychometrics already did it long before. I’ve noticed this a few times. Once, about ten years ago, I was at a conference where computer scientists were talking about some pretty elaborate statistical models, and I realized these were the same as some things I’d seen Iven Van Mechelen and his colleagues working on in the Psychology Department at Leuven. Then, more recently, I wrote this article with David Park on splitting a predictor into three parts, and it turned out that similar work had been done in 1928! by psychometric researcher T. L. Kelley (and, oddly enough, E. Cureton in 1957).