Carol Nickerson investigates an unfounded claim of “17 replications”

Carol Nickerson sends along this report in which she carefully looks into the claim that the effect of power posing on feelings of power has replicated 17 times. Also relevant to the discussion is this post from a few months ago by Joe Simmons, Leif Nelson, and Uri Simonsohn.

I am writing about this because the claims of replication have been receiving wide publicity, and so, to the extent that these claims are important and worth publicizing, it’s also important to point out their errors. Everyone makes scientific mistakes—myself included—and the fact that some mistakes were made regarding claimed replications is not intended in any way to represent a personal criticism of anyone involved.

26 thoughts on “Carol Nickerson investigates an unfounded claim of “17 replications”

  1. “Perhaps the easiest way to convince yourself is by scanning the literature of soft psychology over the last 30 years and noticing what happens to theories. Most of them suffer the fate that General MacArthur ascribed to old generals—They never die, they just slowly fade away.

    (…)

    But in fields like personology and social psychology, this seems not to happen. There is a period of enthusiasm about a new theory, a period of attempted application to several fact domains, a period of disillusionment as the negative data come in, a growing bafflement about inconsistent and unreplicable empirical results, multiple resort to ad hoc excuses, and then finally people just sort of lose interest in the thing and pursue other endeavors.”

    https://en.wikiquote.org/wiki/Paul_E._Meehl

    • “I’m dumbfounded as to why Psychological Science continues to devote space to articles on power posing. Let’s move on already.”

      Perhaps this power-pose thing is an interesting meta-scientific experiment/situation in light of Meehl’s quote above:

      1) Carney, Yap, Cuddy publish power-pose paper
      (Meehl: “period of enthusiasm about new theory” ?)

      2) Several studies/”conceptual replications” concerning “power-posing” in the following years

      (Meehl: “period of attempted applicatoin to several fact domains” ?)

      3) Failed replications (e.g. Ranehill et al, 2015) + Special issue “Comprehensive Results in Social Psychology” (2017) that, if i remember correctly, failed to show any effect accept some felt power

      (Meehl: “period of disillusionment as the negative data come in)

      4) “It was all about felt power all along anyway” + new “p-curve” + datacolada 66

      (Meehl: “growing bafflement about inconsistent and unreplicable empirical results” & “Multiple resort to ad hoc excuses”

      5) Present moment/it remains to be seen what will happen

      (Meehl: “People just sort of lose interest in the thing and pursue other endeavors”

      Now, i find this whole process interesting from a meta-scientific perspective, because you could perhaps see it as a mini-cyle of what Meehl described, and what has probably happened with many psychological theories/phenomena, but concerning power-posing in a relatively short time.

      Now i have been thinking about how to possibly stop this process Meehl described. For this to happen, i reasoned the following needs to happen:

      1) No more low-quality research + publication bias after large scale replications/high quality work: otherwise the cycle will start again/never stop.

      2) Possible follow up work (e.g. to see if felt power might really be a thing) needs to happen soon, and be clearly communicated so researchers working on the specific theory/effect know about it

      3) There needs to be a certain point in which researchers say “enough is enough, this is a dead end at this point in time”. Should people want to pick up the work in 10 years time, they need to know a) where to start, and b) follow high standards (see 1 above).

      Now, for all of this to happen i thought of a format/idea which i have linked to on this blog a few times before. Again: i don’t know if it makes any sense, but it’s the best i could come up with to try and stop the devestating process of psychologcal research Meehl described.

      It (also) takes account of 1, 2, and 3 above, for instance via the name of the publications coming from this format which could be called “power-pose round 1”, “power pose round 2” etc for a) clear communication, and b) a clear possible pick-up point for researchers in the future.

      Here is that format/idea: http://statmodeling.stat.columbia.edu/2017/12/17/stranger-than-fiction/#comment-628652

      • Anon:

        Also in the mix is the interplay between scientific journals, social media, and the traditional news media. Some of this can be seen in Nickerson’s recounting of the power pose story: From one direction, the prominence of power pose in the media (Ted talk, NYT reviews, NPR appearances, etc.) supply some justification for psychology journals to continue publishing papers on the topic; from the other direction, bits and pieces of the research get spread on social media. It’s frustrating how misconceptions can bounce back and forth between the scientific literature and the popular press.

        In Meehl’s day, the media environment was different, but perhaps the relevant outside variable was the potential for ideas from research psychology to be used in therapy or taught in textbooks?

        • “In Meehl’s day, the media environment was different, but perhaps the relevant outside variable was the potential for ideas from research psychology to be used in therapy or taught in textbooks?”

          I actually think the individual might be better off the way things are going now: the internet allows for lots of people to find scientific information themselves, read papers, read blogs, decide who to trust and why, etc.

          Compared to Meehl’s time, i reason scientists today still do what they do (“fake”/bad scientists aren’t a new thing i reason), the media still does what it does (“fake” news/manipulation isn’t a new thing i reason), but the thing that perhaps changed compared to Meehl’s time is that the individual has way more accesss to information, way more power to join the discussion, and way more power to choose.

      • As to follow up, I found a special journal article somewhere a while back where they just invited a bunch of people to replicate the power posing thing. And as I recall, the self-reported confidence kept replicating, while the effects on cortisol levels and the behavioral effects kept not replicating. I didn’t get the impression that the authors were aware of the p-curve results.

        • So based on that, I would say that the self-report finding is real. This leaves the question of whether people are humoring the researcher or if they really do feel more confidence. Either is plausible.

        • Maybe you are referring to the special issue on power posing by the journal “Comprehensive Results in Social Psychology”?

          Leaving aside if that’s the case or not, if i remember correctly, they didn’t give the reader access to the pre-registration (also see Hardwicke & Iaonnidis, 2018 “Mapping the universe of Registered Reports”; https://osf.io/5a63g/).

          But assuming all went as it should, you could view that collection of power-posing papers in that special issue as maybe the most optimal collection of papers on power-posing because 1) there is no publication bias, and 2) there is minimal data-analysis and -gathering flexibility.

          Leaving aside the poor state of the “theory” behind it (i reason it’s poor because if i remember correctly no new significant findings/moderators/etc. were found), this could be a nice starting point for the format/idea presented above.

          I would find it interesting if “Comprehensive Results in Social Psychology” would do another power-posing special issue: Round 2 (but then of course provide the reader with the crucial pre-registration information).

          Let researchers (perhaps some of the same that participated in the 1st special issue) come up with new ideas based on the 1st special issue (or as i would call it “round”), which seems to suggest that it would be useful to perhaps focus on felt-power.

          Let them come up with ways to test how long this felt-power possibly lasts, whether there are negative effects after power-posing and situations of failing, test felt power in different ways, etc.

          To me, that is beginning to look like a real research program.

          But what do i know.

        • Anon:

          My suggestions for the power pose research program are here (scroll down to “A way forward”). I think that it would be necessary to have much more detailed data on individuals.

        • From that blog-post:

          “The idea is to take the positive aspects of the work of Cuddy and others—the inspirational message that rings true for millions of people—and to investigate it using a more modern, data-rich, within-person model of scientific investigation. That’s the sort of thing that should one day be appearing in the pages of Psychological Science”

          I read that post at the time, and was not sure how to interpret the quote above (given most of the rest of the text written in “A way forward”).

          I am still not sure how to interpret it.

          Do you suggest to investigate power-pose and/or felt power via:

          1) experimental within-subject designs or via, what i think is called,

          2) “ethnographic research” which if i understood things correctly is a method of data collection that entails examining the behaviour of the participants in a certain specific social situation and also understanding their interpretation of such behaviour (https://en.wikipedia.org/wiki/Ethnography)

          If it’s 1) experimental within-subject designs, perhaps that fits nicely with investigating felt-power, possible negative effects of power-posing, how long felt-power lasts, etc.

          Perhaps within-subject designs would fit nicely with a new set of pre-registered, non-file drawered, power-pose studies in a possible power-pose special issue no. 2.

          If it’s 2) i am not sure what to do with that…

        • Anon:

          I’m not saying I recommend that people investigate power pose. I don’t have a strong opinion on that. I’m saying that if people want to investigate power pose, I think they should be doing a lot of careful observation on each person. What you’re calling “ethnographic research” (observing people “in the wild”) would be part of that research program; another part would be experimental research, having people try different poses, asking them how they feel, setting up mock job interviews, whatever.

          The part of the power pose story I find interesting is not the original paper (which, after all, was completely disowned by its first author) but rather all the testimonials, people saying how power pose really worked for them. It would be good to have a sense of what that means, how it’s working, how long it lasts, etc. Experiments could help here. But I really think researchers have to start over here: the model of push-a-button, take-a-pill and get statistically significant results, that’s not working.

          Anyway, it’s not up to me. I’m not trying to tell anyone what to do; I’m just trying to offer some help based on my experience as a statistician. There’s been a lot of chasing of noise. Preregistration is a step forward, but I don’t think it’s not going to be so helpful in making progress unless there’s some more serious thinking about measurement. I suspect that one source of this “17 replications” debacle is the goal of proving, once and for all, that the treatment has an effect. But I don’t see that as being so interesting: All sorts of things have effects. Establishing nonzero average effects, that’s the hypothesis testing game but I don’t see it as a generally productive way to do science.

        • I’d also be interested in studying the people that swear power posing helps them.

          James Heathers was able to find a bunch of people who can control their goosebumps, so it’s possible there’s a subset of people out there who become extremely powerful after adopting a power pose. Who knows, maybe their testosterone goes through the roof.

          I’m being serious here, although there’s absolutely no published evidence that power posing does anything, the fact that so many people around the world have apparently tried power posing may have identified a small cohort where this actually works. Studying who these people are and what they have in common would inform who should be doing power posing.

          Unfortunately, from social media it appears pretty obvious who this group of people is that claims power posing helps them, and it seems pretty obvious it’s just a placebo effect.

        • Hi Austin Fournier,

          I believe that you are referring to the special issue of COMPREHENSIVE RESULTS IN SOCIAL PSYCHOLOGY devoted to power posing. The issue contained 10 articles: an introduction, 7 empirical articles examining various outcome variables, a Bayesian meta analysis, and a summary. Of the 7 empirical articles, 6 examined the relation between power posing and feelings of power. Of those 6, 3 found no significant relation between power posing and feelings of power, 1 found a significant relation, and 2 had mixed results (different effects for the overall group of study participants and subgroups of those participants, e.g., males and females). The empirical studies showed almost no evidence for other (behavioral or hormonal) outcomes.

          The editors of the special issue raised the question of whether significant relations between power posing and feelings of power could be due to demand effects.

          Alas, this special issue seems to be behind a paywall for a lot of folks.

          Carol

        • I suspect the kinds of research that Anon and Andrew are discussing are not likely to be interpretable, precisely because the “power pose” phenomenon has become so well known (e.g., Ted Talks) and discussed — the result being that “contamination” and “demand characteristics” are likely to affect the results.

        • Martha:

          Yes, this is a contentious point. The recent Psychological Science article argues that the results cannot simply be explained by a demand effect, but I (and others) don’t find their dismissal of the demand effect to be so convincing. I think that, to the extent people really care about power pose, it will be appropriate to study it in the presence of a demand effect. After all, the demand effect is not new: many of the power pose experiments compare to slouch-like positions, and we’ve been told for a long time not to slouch.

    • Hi Anonymous 2,

      It is PSYCHOLOGICAL SCIENCE’s policy to offer the authors of a critiqued article the opportunity to reply to that critique. The new article by Cuddy, Schultz, and Fosse (2018) is a reply to the critique by Simmons and Simonsohn (2017) of the review article by Carney, Cuddy, and Yap (2015).

      Carol

  2. Hi, all:

    Yes, the CRSP special issue found virtually no behavioral, hormonal, etc. effects. When looking at felt power, the Bayesian meta-analysis did yield “very strong evidence.” It also revealed some moderation that was consistent with the idea (raised by us in the SI and others online) that this reflects some sort of demand characteristic. In any case, there was no evidence that the felt power effect mattered for any actual behaviors, cognition, or physiology.

    There are (at least) two options: You either somehow argue that feeling powerful is really really important even if it doesn’t matter for any other observed behavior (which seems odd to me), or you do the kind of research program that Andrew suggests in which for some people perhaps feeling powerful really matters over time, but there’s no large-scale group effect that would benefit everyone who does it.

    The problem with this latter approach is that it’s useful for Cuddy only insofar as it has no costs for people either (at least if she wants to keep giving interviews, selling books, etc. to the public at large). If the positive effects for some are balanced out by negative effects for others, that’s not great. We (Cesario & Johnson, SPPS, 2018) suggest that this is not an unreasonable possibility given that feeling powerful and confident *when you actually lack important skills for success* can be detrimental over time.

    (Also missing from Cuddy’s ‘this has been replicated so many times’ is the fact that we found no effects on negotiation performance, which one would hope you could find given the claimed applications to negotiating with your boss, etc.: Cesario & Johnson, SPPS)

    In any event, you’re right that some of the articles in the special issue are behind a paywall, which I can’t change for now. If you email me ([email protected]) I’m happy to send you the full special issue.

    • Have you read these 2 comments in this thread:

      1) http://statmodeling.stat.columbia.edu/2018/04/20/carol-nickerson-investigates-unfounded-claim-17-replications/#comment-711458

      2) http://statmodeling.stat.columbia.edu/2018/04/20/carol-nickerson-investigates-unfounded-claim-17-replications/#comment-711571

      Any chance of CRSP issuing a call for a 2nd special issue on power posing building on this “felt power”?

      (Then of course with links to the pre-registration information in the actual paper so the reader can actually check this crucial information: what have you folks been doing/thinking leaving those out?!)

      • Hi, Anonymous:

        *I* will not be doing any more power pose research anytime soon. In the intro to the SI we note that if people want to keep pushing power poses as having positive impact on one’s life, they need to figure out how/when/under which conditions felt power relates to actual behavior. I’m happy to leave it at “possibly really interesting effects but no evidence for it yet” and let those who care more about the topic figure out what that evidence might be.

        If you or anyone else wants to propose a second SI on the topic, I’m happy to review it. I could imagine that it would get accepted if the proposal had some real follow up to the first one and wasn’t just more of the same.

        Also, we’ve been working on the problem of getting the IPAs (which function as a record of the accepted preregistration) online and connected to the published article. That said, if the final published study deviates from the IPA in any way the authors *must* include a section explaining this.

    • Hello, Joe Cesario:

      Thank you very much. I’ll be writing a couple of more statistically sophisticated critiques of this power-posing research. I’m currently slogging my through the 90+ cases in the database for the recent Cuddy, Schultz, and Fosse (2018) p-curving article published in PSYCHOLOGICAL SCIENCE.

      Carol

  3. For those still interested in this saga there is this: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3198470
    In the manuscript I outline what I see as serious errors in the Cuddy et al. p-curve analysis including: an ability of most of the studies to distinguish between a positive effect of power posing versus negative effects of slouching; interpreting two p-values that show a negative effect of power posing as indicating a positive effect of power posing; the inclusion of incorrect p-values, and clear evidence that any overall effect of posture is not driven so much by a positive effect of power posing as by a negative effect for slouching.

Leave a Reply

Your email address will not be published. Required fields are marked *