We’ve all had that experience of going purposefully from one hypothesis to another, only to get there and forget why we made the journey. Four years ago, researcher Daryl Bem and his colleagues stripped this effect down, showing that the simple act of obtaining a statistically significant comparison induces publication in a top journal. Now statisticians at Columbia University, USA, have taken things further, demonstrating that merely imagining a statistically significant p-value is enough to trigger increased publication. . . .
The new study shows that this event division effect can occur in our imagination and doesn’t require literally seeing a pattern that reflects the general population. . . .
OK, I guess at this point you’ll want to see the original, a news article called “Imagining walking through a doorway triggers increased forgetting,” by Christian Jarrett in the British Psychological Society Research Digest:
We’ve all had that experience of going purposefully from one room to another, only to get there and forget why we made the journey. Four years ago, researcher Gabriel Radvansky and his colleagues stripped this effect down, showing that the simple act of passing through a doorway induces forgetting. Now psychologists at Knox College, USA, have taken things further, demonstrating that merely imagining walking through a doorway is enough to trigger increased forgetfulness. . . .
The new study shows that this event division effect can occur in our imagination and doesn’t require literally seeing a doorway and passing through it. . . .
Yes, I do find this funny. But, at the same time, I recognize that these are not easy questions. And, in particular, Jarrett is in a difficult position in that to some extent his job involves the promotion of psychology research, not just the evaluation.
I sometimes have a similar problem when blogging for the Monkey Cage political science blog. A bit of criticism of political science research is OK, but too much and I get pushback.
So, back to the research in question, Lawrence, Z., & Peterson, D. (2014). Mentally walking through doorways causes forgetting: The location updating effect and imagination Memory, 1-9 DOI: 10.1080/09658211.2014.980429.
Based on Jarrett’s description, I see a lot of red flags:
1. Lack of face validity. “Mentally walking through doorways causes forgetting”?? According to Jarrett, “The group who’d imagined passing through a doorway performed worse at the task than the first group who didn’t have to go through a doorway.” This could be true—all things are possible—but it sounds a little weird. And the researchers themselves seem to agree with me on this; see next point.
2. Claims that the effect is both expected and surprising. On one hand, “This effect of an imagined spatial boundary on forgetting is consistent with a related line of research that’s shown forgetting increases after temporal or other boundaries are described in narrative text.” On the other hand, the researchers write, “That walking through a doorway elicits forgetting is surprising because it is such a subtle perceptual feature . . . that simply imagining such a walk yields a similar result is even more surprising . . .”
This is what, following up on some observations from Jeremy Freese, we’ve called the scientific surprise two-step.
3. Lots of different small-n studies but no preregistered replications that I see. Lawrence and Peterson’s finding follows up on a paper by a couple of other researchers, four years ago, which, according to Jarrett, “shows that the simple act of walking through a doorway creates a new memory episode.” The recent paper and that earlier paper have a bunch of studies and comparisons, but it seems like a bit of a ramble (or what Freese calls “Columbian Inquiry”). Each time something interesting shows up, the researchers follow up with a new study that is evaluated in its own way with various idiosyncratic data-analysis choices.
Put it all together, and all I can say is: I’m not convinced. I’m not saying I’m sure these claims are wrong; I just think they’re pretty much at the same status as Nosek, Spies, Motyl’s “50 shades of gray” findings:
Participants from the political left, right and center (N = 1,979) completed a perceptual judgment task in which words were presented in different shades of gray. Participants had to click along a gradient representing grays from near black to near white to select a shade that matched the shade of the word. We calculated accuracy: How close to the actual shade did participants get? The results were stunning. Moderates perceived the shades of gray more accurately than extremists on the left and right (p = .01)
That is, before Nosek et al. tried their own preregistered replication:
We could not justify skipping replication on the grounds of feasibility or resource constraints. . . . We conducted a direct replication while we prepared the manuscript. We ran 1,300 participants, giving us .995 power to detect an effect of the original effect size at alpha = .05.
And got this:
The effect vanished (p = .59).
P.S. Just to be clear, I’m not trying to pick on Christian Jarrett. It’s not his job to evaluate the strength of claims that have been published in refereed psychology journals. We all just need to be aware that you can’t believe everything you see in the papers.