Another failed replication of power pose

Someone sent me this recent article, “Embodying Power: A Preregistered Replication and Extension of the Power Pose Effect,” by Katie Garrison, David Tang, and Brandon Schmeichel.

Unsurprisingly (given that the experiment was preregistered), the authors found no evidence for any effect of power pose.

The Garrison et al. paper is reasonable enough, but for my taste they aren’t explicit enough about the original “power pose” paper being an exercise in noise mining. They do say, “Another possible explanation for the nonsignificant effect of power posing on risk taking is that power posing does not influence risk taking,” but this only appears 3 paragraphs into their implications section, and they never address the question: If power pose has no effect, how did Carney et al. get statistical significance, publication in a top journal, fame, fortune, etc.? The garden of forking paths is the missing link in this story. (In that original paper, Carney et al. had many, many “researcher degrees of freedom” which would allow them to find “p less than .05” even from data produced by pure noise.)

It’s also not clear what makes Garrison et al. conclude, “We believe future research should continue to explore eye gaze in combination body posture when studying the embodiment of power.” If power pose really has no effect (or, more precisely, highly unstable and situation-dependent effects), why is it worth future research at all? At the very least, any future research should consider measurement issues much more carefully.

Perhaps Garrison et al. were just trying to be charitable to Carney et al. and say, Hey, maybe you really did luck into a real finding. Or perhaps psychology journals simply will not allow you to be explicitly say that a published paper in a top journal is nothing but noise mining. Or maybe they thought it more politically savvy to state their conclusions in a subtle way and let readers draw the inference that the original study by Carney et al. was consistent with pure noise.

The downside of subtle politeness

Whatever the reason, I find the sort of subtlety shown by Garrison et al. to be frustrating. For people like me, and the person who sent the article to me, it’s clear what Garrison et al. are saying—no evidence for power pose, and the original study can be entirely discounted. But less savvy readers might not know the code; they might take the paper’s words literally and think that “social context plays a key role in power pose effects, and the current experiment lacked a meaningful social context” (a theory that Garrison et al. discuss before bringing up the “power posing does not influence risk taking” theory).

That would be too bad, if these researchers went to the trouble of doing a new study, writing it up, and getting it published, only to have drawn their conclusions so subtly that readers could miss the point.

Mulligan after mulligan

You may wonder why I continue to pick on power pose. It’s still one of the most popular Ted talks of all time, featured on NPR etc etc etc. So, yeah, people are taking it seriously. One could make the argument that power pose is innocuous, maybe beneficial in that it is a way of encouraging people to take charge of their lives. And this may be so. Even if power pose itself is meaningless, the larger “power pose” story could be a plus. Of course, if power pose is just an inspirational story to empower people, it doesn’t have to be true, or replicable, or scientifically valid, or whatever. From that perspective, power pose lies outside science entirely, and to criticize power pose would be a sort of category error, like criticizing The Lord of the Rings on the grounds that there’s no such thing as an invisibility ring, or criticizing The Rotter’s Club on the grounds that Jonathan Coe was just making it all up. I guess I’d prefer, if business school professors want to tell inspirational stories without any scientific basis, that they label them more clearly as parables, rather than dragging the scientific field of psychology into it. And I’d prefer if scientific psychologists didn’t give mulligan after mulligan to theories like power pose, just because they’re inspirational and got published with p less than .05.

I don’t care about power pose. It’s just a silly fad. I do care about reality, and I care about science, which is one of the methods we have for learning about reality. The current system of scientific publication, in which a research team can get fame, fortune, and citations by p-hacking, and then even when later research groups fail to replicate the study, that even then there is the continuing push to credit the original work and to hypothesize mysterious interaction effects that would manage to preserve everyone’s reputation . . . it’s a problem.

It’s Ptolemy, man, that’s what it is. [No, it’s not Ptolemy; see Ethan’s comment below.]

P.S. I wrote this post months ago, it just happens to be appearing now, at a time in which we’re talking a lot about the replication crisis.

20 thoughts on “Another failed replication of power pose

  1. Why the knock on Ptolemy? His epicicyle model made predictions verifiable with the measurement methods of his time. There will be no Kepler to update the power pose.

    Epicyclical motion is used in the Antikythera mechanism, an ancient Greek astronomical device for > compensating for the elliptical orbit of the Moon, moving faster at perigee and slower at apogee than circular orbits would, using four gears, two of them engaged in an eccentric way that quite closely approximates Kepler’s second law. (https://en.wikipedia.org/wiki/Deferent_and_epicycle)

    As an indication of exactly how good the Ptolemaic model is, modern planetariums are built using gears and motors that essentially reproduce the Ptolemaic model for the appearance of the sky as viewed from a stationary Earth. (http://www.polaris.iastate.edu/EveningStar/Unit2/unit2_sub1.htm).

    (Markdown formatting help for these blockquotes would be welcome.)

  2. Actually, what Ptolemy had was a universal approximator. He just didn’t know it. With enough epicycles and cycles, one could approximate any orbital system as closely as one wanted. In essence, the mistake (but he couldn’t have known it) was to try to construct a theory to fit the details of the approximation machinery.

    Fuzzy logic rules and neural nets are other universal approximators. I wouldn’t go around trying to construct significance into their details, either.

      • The difference is that the model of Ptolemy produced useful predictions, so one could observe that it is working. Dark matter OTH seems to be used to explain why things do not work exactly like we expect, but I do not think it makes useful predictions. At least, that was the case when I first heard about it. Maybe physics has advanced since then.

  3. Relatively new to reading this blog, but given your interests (science, statistics, and a little baseball) you might enjoy something I wrote about all three a year ago: http://www.fangraphs.com/community/stop-thinking-like-a-gm-start-thinking-like-a-player/ . At the time I had no idea how deep the problems in social sciences were, but there are a number of conceptual similarities and errors, especially those revolving around how to maintain scientific rigor with statistical data

    • Matt:

      What you wrote is interesting, but is it really true that sabermetrics has nothing to offer the player? I had the impression that somewhere in sabermetrics there was advice about whether to swing at the first pitch, where to stand in the field, and other things that affect one’s play. And maybe as data become better and more pitch- and swing-specific, sabermetrics can offer insights into what pitches to throw in which scenarios?

      • There are a couple of way in which sabermetrics have helped players. The biggest one was the realization that drawing walks is a highly valuable skill. Internalizing that as a player definitely does change your approach to hitting. Also, in the last 2-3 years there has been a lot of work done with pitchf/x data to try to optimize pitch sequencing. That’s a field of study that can/will have lessons that will be useful down to the high school level, but it’s very new. Outside of that, the places where sabermetrics is valuable to players requires skill levels and data availability that doesn’t exist until high minors to MLB level. Fielding shifts are a good example of this- defenses nowadays will shift around to optimize their positioning for each individual hitter. This has definitely represented a shift in gameplay, but you don’t get the player-level data necessary to do it until essentially the MLB level. Even top-quality D1 college programs do not have the spray charts & such on their opponents to do this sort of positioning. Likewise with any data involving pitchf/x- only in the last 2-3 years has the equipment been installed at AAA or AA levels. I expect that more player-useful data will start reaching lower levels of baseball in the near future, and the piece was in part an encouragement to start thinking that way.

        To give some perspective, a 24-year old rookie in MLB is probably getting a sabermetric/data-based scouting report on his opponent for the first time in his life. That will definitely inform and change they way he does things, but on the other hand, a 24-year old rookie has likely been playing high quality intensive baseball for 12 years and been a professional for 3. The relative contribution of sabermetrics to his whole skillset is fairly minor.

    • Matt,

      I agree with you that SABR doesn’t have much to say about the mechanics of the physical tasks, but as Andrew mentioned, a careful analysis can clearly help for in-game player decision making, especially with regard to a more systematic scouting of opponent tendencies. For example, when studying film, an analyst can team up with a player-expert when comes to coding the data. And then help with the analysis, and get input from the player. This requires that the analyst respect the player’s subject matter expertise, and not write the player off as a dumb jock.

      Overall though, your perspective is dead-on. This is the exact same thing that divides practitioners (players and coaches) and analysts (SABR, academic, etc.) when it comes to the hot hand (e.g. Phil Jackson doesn’t know what he is talking about)

      As you rightly point out, sports analytics folks don’t like to discuss issues like measurement error. Instead, they just throw it all in the error term, and pretend the real phenomenon doesn’t exist. Hey, if a player doesn’t have a higher fg% after hitting a single shot, then he couldn’t possibly have the hot hand! um…

      • Hey Matt

        Somehow my comment got pushed down to the bottom. When I commented I hadn’t finished reading your article to the end, now I have. Wow, your article is great, inspiring actually.

  4. I assumed that the wishy-washy conclusions were the result of referees, so I went looking for a working paper version. But I did find this blog post: “EMBODYING POWER? MORE EVIDENCE THAT POWER POSING DOES LITTLE TO ALTER THE INTRAPSYCHIC EXPERIENCE OF POWER” at http://www.spsp.org/blog/embodying-power

    Two quotes:

    “Why didn’t power posing influence our measures of power? There are a number of possibilities. The one we favor is that the simple power pose is not as impactful as was commonly assumed.”

    ” In the meantime, it seems prudent to be skeptical of the intrapsychic consequences of adopting an expansive posture.”

    These quotes read as very different from the article. I’d guess that the post represents the true opinions of the authors, and the article is a result of what the referees and/or the editor wanted.

    • Will left out what’s between his two quotes. Here’s the whole quote from the blog by Garrison and Schmeichel:

      “Why didn’t power posing influence our measures of power? There are a number of possibilities. The one we favor is that the simple power pose is not as impactful as was commonly assumed. That’s not to say that power poses and body manipulations more generally lack psychological consequences, but rather that their effects are smaller than has been previously estimated in the literature. If this is the case, then more powerful manipulations and more sensitive outcome measures will be needed to detect them.

      The embodiment of power is an interesting and important idea that deserves careful scientific study. Our well-powered, preregistered experiment contributes to research on power posing because it gives an idea of the limits of the phenomenon. As evidence accumulates both in support of and against a hypothesis, the scientific community can get a more complete picture of the phenomenon being studied. In the meantime, it seems prudent to be skeptical of the intrapsychic consequences of adopting an expansive posture.”

      This makes it less clear that they intended what Will claims — although it is still possible that they were trying to be inoffensive by including the parts omitted by Will.

    • Yes, indeed that has been my experience. I had some very strong conclusions based on a re-analysis using proper statistical tests, but was told in very specific terms by reviewers and editors that it needed to be reframed as an “alternative and complementary” analysis to be publishable. …..

  5. Wait, they did find an effect of power pose!

    “The observed effects of power posing on subjective feelings of power were also inconsistent with the findings of both Carney et al. (2010) and Ranehill et al. (2015). We found that adopting a more expansive pose reduced feelings of power compared to adopting a more contractive pose, whereas both Ranehill et al. and Carney et al. found that high-power posers felt subjectively more powerful than low-power posers.” (p. 6, Discussion; statistical analysis in last paragraph on p. 3)

  6. And speaking of mulligans:

    If you check the PGA web site for the origin of the term “mulligan” (see http://www.pga.com/news/golf-buzz/how-mulligan-got-its-name) at the very bottom they have this quote:

    Thus, a “Mulligan” found its niche along in our culture. Its popularity thrives because of who we are – lovers of a good story and a term that somehow fits. It thrives as we are reminded in a classic line from the 1962 John Ford Western film, “The Man Who Shot Liberty Valance.”

    “This is the West, sir. When the legend becomes fact, print the legend.”

    Power pose. Exactly.

Leave a Reply to Z Cancel reply

Your email address will not be published. Required fields are marked *