Skip to content
 

A psychology researcher uses Stan, multiverse, and open data exploration to explore human memory

Under the heading, “An example of Stan to the rescue, multiverse analysis, and psychologists trying to do well,” Greg Cox writes:

I’m currently a postdoc at Syracuse University studying how human memory works. I wanted to forward a paper of ours [“Information and Processes Underlying Semantic and Episodic Memory Across Tasks, Items, and Individuals,” by Gregory Cox, Amy Criss, William Aue, and Pernille Hemmer] that is soon to appear in the Journal of Experimental Psychology: General in which we use Stan to help us analyze correlations among a variety of experimental tasks commonly used to study human memory, thereby helping us understand not only what those tasks tell us about the humans doing them but also how humans use the stimulus materials (words, in this case) across these different tasks. My purpose in forwarding this, beyond the aforementioned shameless plug, is that I believe this project can serve as an example of the utility of several practices you and your blogmates have long advocated:
1) Stan
2) Multiverse analysis
3) Open data exploration

Cox elaborates:

1) Stan

In my own work, I have bounced around between various off-the-shelf samplers and needing to write my own for certain special-purpose models. But Stan made it so easy to implement big multivariate distributions and rather complex link functions (the Wiener first passage time distribution) and NUTS explores the posterior in such an efficient manner that I find it hard to imagine I will go back to other programs or custom samplers any time soon. We might have been able to do the work without Stan, but we would have had to either reduce the scope of our project or rest uneasy in the worry that our posterior samples were just a cloud of dust. Stan makes me feel that we can rest easy.

2) Multiverse analysis

Like anyone who analyzes data, particularly in an exploratory setting (see below), we tried a bunch of different things before settling on the stuff in the main text. I admit, we originally were not going to talk about all those other tries in the paper until the editor asked about them, but I’m very glad he (the editor) did! For context, our analyses involved selecting a certain number of principal components to focus on, so that we could reduce the complexity of the data to something we might have a chance of understanding (and communicating to a reader), but how was this choice made? While hardly gripping reading, Appendix D shows the results of different choices, and even the results that would occur if individual tasks were dropped out (as if we had not measured them). The result is that a reader has more information to make a reasoned judgment about our analyses, since they now have a clearer picture of why we thought the main one was most useful and also how much the conclusions might change as a function of different choices or different observations.

3) Open data exploration

The original motivation for this study, for which credit goes to Pernille Hemmer and Amy Criss, was to produce an openly available set of data across these various tasks to aid the development of memory theory, so really this whole project was “born free”. We have tried to adhere to this ideal as best we can, putting out our data and code on the OSF, doing so as soon as we submitted it for publication. We know that there is a lot more detail in these data than we have analyzed so far, but we are finite creatures. By making all of our data/methods available, we don’t have to do all the work ourselves, so this is a benefit to us–someone with a good idea can just do it and contribute immediately without going to the trouble of collecting another 450 participants. And obviously it is a benefit to those folks with good ideas since now they can test and refine those ideas much more easily. As we say in our discussion, “we do not expect this will be the final word with regard to this dataset or the tasks represented therein.”

But while I think our work is an example of a number of good research practices, I am sure that there are things we could do better . . . As a final remark, many psychologists/cognitive scientists like myself feel frustrated that the public face of our field is one that is pocked with a mixture of tabloid silliness, deliberate ignorance, and outright fraud, as discussed often on your blog. Thus I hope that our work, as imperfect as it is, can also serve as a reminder that there have been a lot of us—going back to the birth of scientific psychology with Helmholtz, Wundt, etc.—that have been diligently trying to understand how minds work through careful experiments, rigorous analyses, and well-developed theories. We may not be the loudest voices in the room, but we were always there and always will be.

Cox adds:

While I believe the public face of psychology is, indeed, unpleasant to look upon, I don’t blame critics for that fact, nor do I mean to impugn the vast majority of psychology researchers who, like most of us, are doing the best we can with what we have. I also don’t mean to say that that ONLY way to do good psychological science is the way we did it—different research questions require different kinds of experiments, analyses, and theories (to say nothing of the fact that we are not perfect ourselves). Rather, I want to emphasize the importance of tight links between experiment, analysis, and theory, and that the three research practices I discuss (Stan, multiverse, and open exploration) make it easier to forge those links and make real progress.

Leave a Reply