Skip to content
 

As if the 2010s never happened

E. J. writes:

I’m sure I’m not the first to send you this beauty.

Actually, E. J., you’re the only one who sent me this!

It’s a news article, “Can the fear of death instantly make you a better athlete?”, reporting on a psychology experiment:

For the first study, 31 male undergraduates who liked basketball and valued being good at it were recruited for what they were told was a personality and sports study. The subjects were asked to play two games of one-on-one basketball against a person they thought was another subject but who was actually one of the researchers.

In between the two games, the participants were asked to fill out a questionnaire. Half of the subjects were randomly assigned questions that probed them to think about a neutral topic (playing basketball); the other half were prompted to think about their mortality with questions such as, “Please briefly describe the thoughts and emotions that the thought of your own death arouses in you” . . .

That’s right, priming! What could be more retro than that?

The news article continues:

The researchers hypothesized that according to terror management theory, those who answered the mortality questions should show an improvement in their second game. When the results of the experiment, which was videotaped, were analyzed, the researchers found out the subjects’ responses exceeded their expectations: The performance in the second game for those who had received a memento mori increased 40 percent, while the other group’s performance was unchanged.

They quoted one of the researchers as saying, “What we were surprised at was the magnitude of the effect, the size in which we saw the increases from baseline.”

I have a feeling that nobody told them about type M errors.

There’s more at the link, if you’re interested.

I feel bad for everyone involved in this one. Understanding of researcher degrees of freedom and selection bias has only gradually percolated through psychology research, and it stands to reason that there are still lots of people, young and old, left behind, still doing old-style noise-mining, tea-leaf-reading research. I can only assume these researchers are doing their best, as is the journalist reporting these results, with none of them realizing that they’re doing little more than shuffling random numbers.

One recommendation that’s sometimes given in these settings is to do preregistered replication. I don’t always like to advise this because, realistically, I expect that the replication won’t work. But preregistration can help to convince. I refer you to the famous 50 shades of gray study.

31 Comments

  1. Reference to the study in question, which is oddly not mentioned in the news article, is:

    Zestcott*, C. A., Lifshin*, U., Helm, P., & Greenberg, J. (2016). He dies, he scores: Evidence that reminders of death motivate improved performance in basketball. Journal of Sport & Exercise Psychology, 38, 470-480.

    PDF: https://www.researchgate.net/profile/Colin_Zestcott/publication/309144536_He_Dies_He_Scores_Evidence_that_Reminders_of_Death_Motivate_Improved_Performance_in_Basketball/links/5877ea9608aebf17d3bbc9c1.pdf

  2. Anoneuoid says:

    I can only assume these researchers are doing their best, as is the journalist reporting these results, with none of them realizing that they’re doing little more than shuffling random numbers.

    That was the most disheartening aspect for me. Otherwise good people get chewed up by the system and put on a path of pseudoscience. There should be NHST rehab, retreats, and support groups.

    Apparently AA has ~ 1.4 million members and uses ~$10 million each year, so it only costs them about ~$7-8 per member each year (this is surprisingly low to me…) .[1] To get an estimate of the number of addicts, around 90k people applied for NIH grants in 2015[1], so that would be $720k per year, or funding a single relatively big lab. This seems totally doable.

    The question is whether these people *want* to be helped though. In my experience the hardest part is getting them to admit there is a problem, especially when they see all their colleagues and coworkers doing the same thing.

    [1] https://www.aa.org/assets/en_US/en_gsofinancialinfo.pdf
    [2] https://nexus.od.nih.gov/all/2016/05/31/how-many-researchers/

  3. Marcus says:

    There is a weird non-disclosure in the paper (unless I missed it). In study 1 participants played basketball against a confederate and their scores represent the DV (post-manipulation) and control variable (pre-manipulation). It seems from at least one account of this study that the confederate was actually the lead researcher! Now, the paper claims that the confederate/researcher was blinded to which condition he was in but this is still a really weird thing to not mention in the paper, especially since he probably got better at playing one-on-one basketball as the study went on.
    https://arstechnica.com/science/2016/11/youre-all-going-to-die-a-scientifically-proven-pep-talk-for-winning/

    Of course there are some many other issue here (low power, garden-of-forking-paths) that this particular issue hardly seems worth mentioning. The number of possible alternative analyses is particularly dizzying. A multiverse analysis would have been nice.

  4. gdanning says:

    “In a second, subtler experiment, participants took part in a timed basket-shooting challenge where the instructions and rules were given by a researcher wearing a T-shirt with a large skull and the word “Death” on the front. The T-shirt was visible to half of the participants (randomly selected) and was covered by a zipped jacket for the others. … Subjects who viewed the skull-and-death shirt outperformed those who did not by 30 percent.”

    Clearly, viewing a guy wearing a jacket suppresses athletic performance, as was apparently known to the Greeks (hence the nude athletes at the Olympics).

    • LemmusLemmus says:

      So much for the researcher being blinded.

    • Jeff says:

      “Participants were randomly assigned to the MS or the control condition. Participants in the MS condition were asked the prototypical open-ended questions regarding their mortality: ‘Please briefly describe the emotions that the thought of your own death arouses in you’ and ‘Jot down as specifically as you can, what you think will happen to you as you physically die and once you are physically dead.’ […] In the control condition, participants responded to parallel questions about playing basketball: ‘Please briefly describe the thoughts and emotions that the thought of playing basketball arouses in you’ and ‘Jot down as specifically as you can, what you think will happen to you as you play basketball.'”

      Clearly, being primed to be aware of your basketball-induced emotions and what’s about to happen to you as you play basketball is going to be pretty distracting the next time you try to play basketball. They said they discarded the actual survey responses but I’d love to see what participants said they thought was going to happen to them.

    • A blood pancake says:

      Does eating blood pancakes remind us of our mortality? And perhaps make us perform better that way… ah, they knew what they were doing when they fed those to us in the elementary school. Bloody pancakes, death metal and Sartre – that’s the cocktail for a productive day. Or at least to becoming a Hot Topic customer.

      Well, anyway. This might’ve been brought up ad nauseam, but it never ceases to amaze me how researchers seem to have such an optimistic view on how they can just “induce” emotional states in people. Let us have a guy wear a t-shirt with a skull – and BAM: people now have been induced to tremble before their mortality. Let us play some stock sad music and BAM – sadness has been induced into the poor participant’s brain.

  5. Ben Prytherch says:

    I just skimmed the paper. Absolutely everything is in there… tiny sample size, participants removed for questionable reasons (“one participant showed up with his girlfriend”), regression analyses using one of many potential DVs and different subsets of a large pool of IVs, difference-between-significant-and-not-significant-is-significant, post-hoc theorizing about why one of their many potential DVs gave a significant result while the others didn’t, post-hoc power analyses showing high power for tests that produced small p-values, heavy use of causal language in the discussion, interpreting p > 0.05 as “no effect”, and unexplained and strange data analysis choices (combinations of bootstrapping w/fully parametric methods; 5,000 bootstrap samples for study 1; 10,000 bootstrap samples for study 2).

  6. Jacob says:

    FWIW, most of the terror management theory stuff I’ve ever seen was far more plausible…things like being reminded of crime in your community may make you more intolerant of outgroups or being reminded of your mortality making you cling to local traditions (e.g., less likely to accept birth control or condoms in cultures that have social taboos about those things).

    • Ok, more plausible on the face of it, but was the actual data and analysis any better? Did it provide evidence of a real effect or just the usual p less than 0.05 with specific test chosen after seeing the data?

    • Sofia says:

      Really? To me, it has always seemed that to explain the results, they (TMT researchers) would not actually need the construct of “mortality salience” at all – that all results could be explained by feelings of uncertainty, lack of control, or simply negative affect (I know they often control for negative affect, but not always). And second, this research line has often used very small samples and complex manipulations, and also, really different manipulations which are considered to have the same effect. For instance, I remember one study in which looking oneself in the mirror was supposed to cause mortality salience via first causing “existential salience” and then, via association between existence and non-existence, mortality salience. These methodological issued have made me somewhat doubtful of this research line. I mean, I can believe that making people feel sort of bad/uncertain may make them more judgmental and, perhaps, to cling to their values and become more communal/less individualistic, but I’m not sure the TMT research has shown that it’s mortality salience that causes these things.

      Granted, I’m a bit out of the “loop” now – I used to follow the TMT research pretty closely 5-10 years ago. It’s possible that the researchers have since answered to these criticisms. And I’m definitely not saying they did anything wrong, just that I don’t find most of the (older) results very plausible.

      • Anoneuoid says:

        I remember one study in which looking oneself in the mirror was supposed to cause mortality salience via first causing “existential salience” and then, via association between existence and non-existence, mortality salience.

        So looking in the mirror is treated the same as recently being beat up and mugged (or in a car accident, or nearly starving lost in the woods, etc)? Surely they do not just say mortality salience is present or not?

        • Sofia says:

          To my shame, I remembered the mirror thing wrong. It’s a separate manipulation; there is a series of studies examining mortality salience x self-awareness interaction effects on outcomes, and looking into mirror (vs. not) is the self-awareness manipulation.

      • mb says:

        ” To me, it has always seemed that to explain the results, they (TMT researchers) would not actually need the construct of “mortality salience” at all – that all results could be explained by feelings of uncertainty, lack of control, or simply negative affect ”

        That is basic the consensus outside of the core TMT group.

      • Jacob says:

        I would agree that there are often plausible alternative explanations for the results that otherwise seem reasonably well-founded. I can’t characterize the literature whatsoever since my knowledge of it is mostly limited to a few job talks and presentations from colleagues. In general I would say it seems difficult to isolate “mortality salience” from other, related constructs and I’m not sure whether/how this area has done so.

        • SF says:

          TMT research has focused more heavily nowadays on the DTA concept – which looks at the number of death-related constructs that become accessible after threat. These findings typically show that MS effects that are commonly reported are the result of increases in DTA. That is, for example, defence of worldviews or bolstering self-esteem decreases DTA after an MS manipulation. In addition, threats to self-esteem and worldviews increases DTA. Some research has been done exploring why DTA increases when exposed to these threats and the evidence again suggests it is not to do with an increase in negative thoughts or anger.

Leave a Reply