I’ve started reading the piece you sent me on Seth. Very interesting stuff. I generally tend to think that one can get useful evidence from a wide variety of sources — as long as one keeps in mind the nature of the limitations (and every data source has some kind of limitation!). Even anecdotes can generate important hypotheses. (Piaget’s observations of his own babies are great examples of real insights obtained from close attention paid to a small number of children over time. Not that I agree with everything he says.) I understand the concerns about single-subject, non-blind, and/or uncontrolled studies, and wouldn’t want to initiate a large-scale intervention on the basis of these data. But from the little bit I’ve read so far, it does sound like Seth’s method might elicit really useful demonstrations, as well as generating hypotheses that are testable with more standard methods. But I also think it matters what type of evidence one is talking about — e.g., one can fairly directly assess one’s own mood or weight or sleep patterns, but one cannot introspect about speed of processing or effects of one’s childhood on present behavior, or other such things.
My thoughts: that’s an interesting distinction between aspects of oneself that can be measured directly, as compared to data that are more difficult to measure.
I remember that Dave Krantz once told me that many of the best ideas in the psychology of decision making had come from researchers’ introspection. That sounds plausible to me. Certainly, speculative axioms such as “minimax risk” and similar ideas discussed in the Luce and Raiffa book always seemed to me to be justified by introspection or by demonstrations of the Socratic-dialogue type that (such as in Section 5 of this paper, where we demonstrate why you can’t use a curving utility function to explain so-called “risk averse” attitudes).
One of the discussants of Seth’s paper in Behavioral and Brain Sciences compared introspection to self-experimentation. Just as self-experimentation is a cheaper, more flexible, but limited version of controlled experiments on others, introspection is a cheaper etc. version of self-experimentation.
Back to Susan’s comments: she appears to agree with Seth that it’s not a good idea to jump from the self-experiments to the big study. So there should be some intermediate stage . . . pilot-testing with volunteers? How much of this needs to be done before he’s ready for the big study? More generally, this seems to be an important experimental design question not addressed by the usual statistical theory of design of experiments.