I contacted Anna Dreber, one of the authors of the paper that failed to replicate power pose, and asked her about a particular question that came up regarding their replication study. One of the authors of the original power pose study wrote that the replication “varied methodologically in about a dozen ways — some of which were enormous, such as having people hold the poses for 6 instead of 2 minutes, which is very uncomfortable.” As commenter Phil put it, “It does seem kind of ridiculous to have people hold any pose other than ‘lounging on the couch’ for six minutes.”
In response, Dreber wrote:
We discuss this in the paper and this is what we say in the supplementary material:
A referee also pointed out that the prolonged posing time could cause participants to be uncomfortable, and this may counteract the effect of power posing. We therefore reanalyzed our data using responses to a post-experiment questionnaire completed by 159 participants. The questionnaire asked participants to rate the degree of comfort they experienced while holding the positions on a four-point scale from “not at all” (1) to “very” (4) comfortable. The responses tended toward the middle of the scale and did not differ by High- or Low-power condition (average responses were 2.38 for the participants in the Low-power condition and 2.35 for the participants in the High-power condition; mean difference = -0.025, CI(-0.272, 0.221); t(159) = -0.204, p = 0.839; Cohen’s d = -0.032). We reran our main analysis, excluding those participants who
were “not at all” comfortable (1) and also excluding those who were “not at all” (1) or “somewhat” comfortable (2). Neither sample restriction changes the results in a substantive way (Excluding participants who reported a score of 1 gives Risk (Gain): Mean difference = -.033, CI (-.100,
0.034); t(136) = -0.973, p = 0.333; Cohen’s d = -0.166; Testosterone Change: Mean difference = -4.728, CI(-11.229, 1.773); t(134) = -1.438, p = .153; Cohen’s d = -0.247; Cortisol: Mean difference = -0.024, CI (-.088, 0.040); t(134) = -0.737, p = 0.463; Cohen’s d = -0.126. Excluding participants who reported a score of 1 or 2 gives Risk (Gain): Mean difference = -.105, CI (-0.332, 0.122); t(68) = -0.922, p = 0.360; Cohen’s d = -0.222; Testosterone Change: Mean difference = -5.503, CI(-16.536, 5.530); t(66) = -0.996, p = .323; Cohen’s d = -0.243; Cortisol: Mean difference = -0.045, CI (-0.144, 0.053); t(66) = -0.921, p = 0.360; Cohen’s d = -0.225). Thus, including only those participants who report having been “quite comfortable” (3), or “very comfortable” (4) does not change our results.
Also, each of the two positions was held for 3 min each (so not one for 6 min).
So, yes, the two studies differed, but there’s no particular reason to believe that the 1-minute intervention would have a larger effect than the 3-minute intervention. Indeed, we’d typically think a longer treatment would have a larger effect.
Again, remember the time-reversal heuristic: Ranehill et al. did a large controlled study and found no effect of pose on hormones. Carney et al. did a small uncontrolled study and found a statistically significant comparison. This is not evidence in favor of the hypothesis that Carney et al. found something real; rather, it’s evidence consistent with zero effects.
In our study, we actually wanted to see whether power posing “worked” – we thought that if we find effects, we can figure out some other fun studies related to this, so in that sense we were not out “to get” Carney et al. That is, we did not do any modifications in the setup that we thought would kill the original result.
Indeed, lots of people seem to miss this point, that if you really care about a topic, you’d want to replicate it and remove all doubt. When a researcher expresses the idea that replication, data sharing, etc., is some sort of attack, I think that betrays an attitude or a fear that the underlying effect really isn’t there. If it were there, you’d want to see it replicated over and over. A strong anvil need not fear the hammer. And it’s the insecure researchers who feel the need for bravado such as “the replication rate in psychology is quite high—indeed, it is statistically indistinguishable from 100%.”
P.S. I wrote the above post close to a year ago, well before the recent fuss over replication trolls or whatever it was that we were called. In the meantime, Tom Bartlett wrote a long news article about the whole power pose story, so you can go there for background if all this stuff is new to you.