Should I be upset that the NYT credulously reviewed a book promoting iffy science?

I want to say “junk science,” but that’s not quite right. The claims in questions are iffy, far from proven, and could not be replicated, but they still might be true. As usual, my criticism is the claim that the evidence is strong, when it isn’t.

From the review, by Heather Havrilesky:

The social psychologist Amy Cuddy was, at the very least, in the right place at the right time when she delivered her 2012 TED talk titled “Your Body Language Shapes Who You Are,” perhaps better known as “that YouTube video about posing like Wonder Woman.” Her central idea was simple: By assuming a pose associated with power, you can actually make yourself feel more powerful before an important job interview or presentation.

Before going on, let me say that the idea that positive thinking, or even positive posing, can make you feel powerful, is not controversial nor is it new. I’ll get to the controversy in a bit.

But for now let me continue from the review:

By adopting the posture and poses of the powerful, we not only boost our mood and improve our outlook, we affect the way other people experience us. This creates a positive feedback loop: Our neurotic mind is quieted, we are more present, we can observe those around us and serve them better, and we manifest this power physically, leading to even more success and satisfaction.

OK, fine. Again, this is all reasonable but it does not capture the science, or attempted science, that made Cuddy more than just another business guru. Here’s the controversial claim, which I can conveniently copy from the Ted talk webpage:

Screen Shot 2016-01-02 at 11.08.13 PM

And here’s Havrilesky again:

While Cuddy’s research seems to back up her claims about the effects of power posing, even more convincing are the personal stories sent to the author by some of the 28 million people who have viewed her TED talk. Cuddy scatters their stories throughout the book, which renders its tone a little uneven at times: Generalizations yield to first-person anecdotes yield to social science research yield to an almost evangelical call to turn off your crazy, negative brain and occupy the moment more fully. Ultimately, though, it’s Cuddy’s presence itself — her openhearted desire to help the insecure and the uneasy in this age of anxiety — that shines through most clearly. Unlike so many similar books aimed at ushering us to our best lives, “Presence” feels at once concrete and inspiring, simple but ambitious — above all, truly powerful.

And that’s the part that is bothering me, which is that the reviewer is giving Cuddy’s research a free pass.

In the world of research psychology, Cuddy’s work is much disputed. For example this, from Eva Ranehill, Anna Dreber, Magnus Johannesson, Susanne Leiberg, Sunhae Sul, Roberto Weber, and Mallor Kidwell:

In a growing body of research, psychologists have studied how physical expression influences psychological processes . . . A recent strand of literature within this field has focused on how physical postures that express power and dominance (power poses) influence psychological and physiological processes, as well as decision making . . . Carney et al. found that power posing affected levels of hormones such as testosterone and cortisol, financial risk taking, and self-reported feelings of power in a sample of 42 participants . . .. We conducted a conceptual replication study with a similar methodology as that employed by Carney et al. but using a substantially larger sample (N = 200) and a design in which the experimenter was blind to condition. . . .

And here’s what they found:

Consistent with the findings of Carney et al., our results showed a significant effect of power posing on self-reported feelings of power. However, we found no significant effect of power posing on hormonal levels or in any of the three behavioral tasks.

So, no. Cuddy’s presence might well be effective—she wouldn’t be much of a motivational speaker without that presence. But presence and an appealing idea and a bunch of personal stories don’t make it real.

No villains here

There are no bad guys in this story.

Havrilesky’s a book reviewer doing her job. Indeed, she expresses some skepticism, not about Cuddy’s science but about the general point of this sort of business self-help book. I do not intend for this post to be a criticism of Havrilesky, as it’s not expected of her to doubt the science claims of a famous Harvard professor. One reason I run posts such as this is to change the climate a bit, to make the Havrileskys of the world less deferential to all the Cuddys out there.

It’s funny—to a certain audience such as myself, terms such as “power pose” and “Ted talk” are laugh lines. Sometimes I have to be reminded that in the larger world this stuff is respected. Heck, in the larger world, Malcolm Gladwell is respected. Stephen Jay Gould too, maybe. Albedo-boy. The fat-arms-and-political-attitudes researchers. David Brooks. OK, maybe not David Brooks, but you get the idea.

And, of course, all these people have done useful work (yes, even David Brooks in his earlier, funnier period). They earned their reputations, then pissed them away.

Anyway, the readers of this blog are ahead of the curve with regard to power pose, himmicanes, and the like. But it will still take awhile for this awareness to suffuse the general educated public.

OK, back to the “no villains” thing.

Cuddy’s a researcher with some strong views. It’s hard to do research without some commitment to your ideas. Yes, I think it would be appropriate for her to respond to the non-replication by toning down her claims and discussing the relevance of her ideas in a world in which some of her results did not hold up. But I haven’t read her book so I’m in no position to criticize her for its contents.

The Ted talk people—ok, maybe they’re bad guys of a sort in that someone must have alerted them to the problems with Cuddy’s work. But I don’t want to slag too much on these guys. After all, I’d gladly give a Ted talk if asked. Sure, I mock them. But I’d love the exposure. I’ve published in PPNAS, too!

Should I be upset at all?

But why should I care at all? Suppose that power posing doesn’t actually alter your body chemistry, that it just makes you feel good cos it makes you feel like it’s making you feel good? What’s so wrong with that?

Indeed, we could try a thought experiment:

Bunny Power

Renowned Harvard professor does controlled trial on the effectiveness of the rabbit’s foot. It turns out that it really works. Holding a rabbit’s foot in your pocket makes you a better negotiator, a better speaker, an all-around more confident person. And hormonal measurements bear this out.

It could be. It’s plausible.

It doesn’t replicate.

So what? If it makes people happy, why not?

I don’t know what to say about this one. As a scientist I don’t like the idea of spinning stories around things that aren’t really happening. And, as a statistician, I don’t like overstating the evidence for various attractive or unattractive speculations. But if this is all giving people confidence, maybe it’s fine. No harm, no foul.

P.S. It’s sooooo satisfying to write this on the second day in January, but scheduling it for the next open slot, which is in Apr. I love not being a slave to the hype cycle.

P.P.S. But noooooo . . . I’m not just a slave to the hype cycle, I’m an active contributor! In the meantime I adapted the above for Slate and also blogged on it a few times. But I never got around to deleting this post. I decided just to keep it as is, as it represents a slightly different take on the story.

64 thoughts on “Should I be upset that the NYT credulously reviewed a book promoting iffy science?

  1. >What’s so wrong with that?

    What’s so wrong is that the placebo effect can be gained for free through other methods. The problem isn’t power poses, it’s the possibility of a “power pose industry” arising, extracting value from people by selling them power pose eBooks, DVDs, iPhone apps, conferences, etc. The value that it extracts from society may wind up being more expensive than the actual value of the placebo.

    • I think you need to think about economics a bit. Extracting money, as you call it, or generating revenue as a company calls it does not disappear from the economy, unless it is a very small percentage of people who hoard that money rather than recirculate it through the purchase of other goods and services.

      If the person who buys the product or service finds no tangible or intangible value in it, then they are unlikely to repurchase the product. They will instead spend their money elsewhere.

      • So it’s OK to fleece the suckers because eventually you’re going to spend the money on something for yourself and that makes it all ok?

        Sorry.

        How about this version: person X buys expensive cancer treatment Y with the savings for his children’s college fund because he believes p < 0.05 that it has a 90% chance of extending his life from 50 years to a nominal 80 or 90 years. It turns out expensive cancer treatment Y is totally ineffective when studied in a properly designed double-blind placebo trial.

        But it's OK to sell this fancy treatment even though it's totally ineffective and the doctors actually know it's ineffective, since the doctors in question plan to buy a Yacht and sail the Caribbean where poor workers at rum distilleries will eventually receive some of this money. And besides the cancer patient will just die and not spend any more money on this treatment.

        It's the same basic fallacy taken to its extreme because it makes the problem clearer. It's not ok to sell things using false statements.

        • My point is that what you call ‘fleece the suckers’ may not actually be viewed as such by the customers.

          Why do you get to decide what someone else values and how they value it?

        • I don’t, but I also think that people who are lied to buy the lie, and they pay a lot more than if they knew the truth. Let’s just suppose Cuddy’s book had a prominent warning on the front “Warning: all of these results were investigated in much larger and better designed studies and turned out to be false”

          How many copies would be sold? How much would be paid? The difference is substantial I think.

          The problem of asymmetric information in economics is well understood. We have laws and entire police squads devoted to fraud because it’s well understood that you can extract money by fraudulent means and it’s unethical and illegal.

          When science becomes part of that problem it’s a serious deterioration of society.

        • I think if this were not done, the healthcare industry would be 1/3 the size it currently is.

          I do not think there is any way to control this type of thing other than by market forces and litigation of fraud, short of a single payer system where treatment options are made centrally.

        • So it’s ok, because 2/3 of doctors do it already?

          If you’re saying that it’s hard to regulate and hard to prevent, then I agree with you. If you’re saying we should just give up because it’s hard, or that we should just accept it because it’s really maybe not that bad… then I totally disagree with you.

        • @Daniel, thanks for this! I had been thinking for a long time about the quackery I see my friends unfortunately engaging in and since it seemed to work for them I always held the philosophy that amounted to “eh, what’s the harm?” But your example is excellent, and makes clear the choice isn’t between treatment X and no treatment, but the allocation of limited resources to one of many possible treatments. If someone chooses (or is tricked into) engaging in an ineffective treatment it diverts their limited resources.

        • The “truth” is a bit less extreme, at least for the magnitude of expected benefits and the likelihood that your oncologist is a future yacht owner. Oncologists do get keep a percentage of the costs of their drugs, and do sometimes give drugs solely on the basis of hope to patients who have cancer types for which there is no proven benefit. Those drugs are on the market on the basis of controlled trial with “p-values < 0.05" for extending the median survival from 6 months on standard therapy to 9 months on new-improved therapy.

          The response by Curious asks why we should think we have a stake in this. It's because his and my health insurance dollars (by way of corporate health plans or taxation) are paying for this extra treatment sometimes on the basis of good evidence for meager to modest measurable benefit and other times on no credible evidence of any benefit. The number of drugs with expected life-prolongation of a year or more is pretty small. Who is getting fleeced? We are. Who screams about health insurer denials and death panels. Not everyone, but a sufficient number of voters to prevent sensible regulation.

  2. > What’s so wrong with that?

    It does terrible, unjustified violence to animals.

    But I guess you could say that of the eating habits of most people in the developed world.

  3. “Cure: A journey into the science of mind over body” is a book by Jo Marchant and has been reviewed very favorably:

    “Ms. Marchant [PhD in microbiology] writes well, which is never a guarantee in this genre… Second, [she] has chosen very moving characters to show us the importance of the research…and she has an equal flair for finding inspirational figures… the studies are irresistible, and they come in an almost infinite variety.” New York Times

    “A cautious, scrupulous investigation of how the brain can help heal our bodies. It is also an important look at the flip side of this coin, which is how brains damaged by stress may make bodies succumb to physical illness or accelerated aging… ‘Cure’ points a way toward a future in which the two camps [mainstream medicine and alternative therapies] might work together. After all, any medicine that makes a patient better, whether conventional, alternative, or placebo, is simply medicine.” Wall Street Journal

    However, despite the glowing reviews, Marchant’s book tends to be merely a celebration of the placebo effect; she strives mightily to find a mystical/scientific connection between mind and body. She travels the world seeking physicians and scientists who have carried out supposed control trials which putatively defy conventional explanations. Just like the old days when a radio preacher would cure countless listeners of “suger” diabetes, financial woes and unfaithful spouses by putting a hand on the radio.
    Nevertheless, she does write in an engaging manner even if she is annoyingly lacking in credulity when sensational results/claims are put forward. On the bright side of things, faith-based medicine is a whole lot cheaper than any alternative. The review in the New York Times hinted at this: “Ms. Marchant has won me over again, with a chapter about the pilgrims of Lourdes.”

  4. “But if this is all giving people confidence, maybe it’s fine. No harm, no foul.”

    I disagree — overconfidence can do harm! (Think overconfident politicians, military leaders, physicians, …). Degree of confidence needs to be appropriate to degree of competence.

      • Thanks! I’ve read the part of the interview about Gould’s “The Mismeasure of Man” and the interview does blame Gould for not doing the measurements correctly (and I see that Wikipedia describes it at some length), but overall Susan Gelman seems to be quite positive about this book. From how you phrased your post I thought that you consider Gould a laughable pseudoscientist or something, on par with those publishing obvious nonsense such as himmicanes or whatnot. But maybe that’s not what you meant.

        • Amoeba:

          I haven’t followed the details but I think the consensus is that Gould knowingly falsified some of the evidence on that measurements thing.

  5. >>>Cuddy’s a researcher with some strong views. It’s hard to do research without some commitment to your ideas.<<<

    That's where I think the academic system has failed us. I don't expect Cuddy to be more introspective and open to criticism, what I care about more is how vocally *other* researchers are responding. When confronted with crappy conclusions one would hope the fellow researchers reading this would create a furor. But we don't see that (apart from exceptions like Andrew).

    The marketplace of ideas fails if academics stay passive in the face of bad ideas. Why is the Psych. Associations silent on Cuddy's work? How about her fellow faculty from her Dept. or the University? If we every time we read a Himmicanes-like work Profs. vocally protested and pointed out the crap, the general public may not be so easily duped.

    That's where I think Andrew's focus on getting the researcher in question to admit their errors is misguided. I hardly care if they do so long as *other* researchers raise such a stink that it no longer becomes natural for TED or NYT to push these story-lines. At least not without as much coverage to the detractors.

    • Rahul:

      I agree. It’s about science as a process, not just individual scientists. And we can imagine a division of labor in which some people do the speculations and other people criticize. That said, I think there’s a real advantage to having both sides of this equation in a single person: I’m a better critic because I work on active research projects, and I think Cuddy etc would be a better researcher if she were also a critic of her own work. To put it another way: Yes, we can rely on outside critics but it would be even better, I think, if researchers recognized internal criticism to be part of the discovery process.

      • Andrew:

        A good dose of internal criticism is indeed good. But like you say there’s an inherent tension between being passionate & committed to ones ideas and that.

        Hence the system works best if I can let a researcher follow his ideas with as much evangelical zeal as he will with the assurance that prompt & harsh criticism will come *externally* from watchful colleagues.

        • Rahul:

          Yes, I agree. Just as an economic safety net is said to be a good idea in that it frees up people to try out risky business ideas, so should a system of vigorous post-publication review allow researchers to be more bold with their speculations.

    • I disagree with Rahul. Researchers need to remain very skeptical of their own work, you can’t just leave it to others. The reason is that many of these researchers are already very well-established and/or from brand-name universities. They can and do publish any old crap in a top journal because the moment the editor and/or reviewer sees their name they assume the researcher must be right. I have even seen papers with not a single “statistically significant” effect appearing in the paper being accepted in top journals just because the author asserted that the effect is there and the name has instant brand recognition, sort of the Levi jeans of scientists.

      Also, the researcher eventually comes to believe the hype about him-/herself; if everyone says he/she is such a superstar, he/she must be right. This reminds me of the joke about us linguists: asked to prove that every odd number is prime, we reason like this: 1 is odd and prime, 3 is odd and prime, 5 is odd and prime, 7 is odd and prime, 9 is…well odd and not prime, but we would lose the generalization that every odd number is prime so let’s just assume it’s prime. This joke is not really a joke, it cuts rather close to the bone, and not just in linguistics. Amy Cuddy also operates on this principle when faced with counterexamples, and it is the reason that people assert statistical significance if p is less than 0.05, and continue to assert it even if p is 0.10 because hey, it’s getting there.

      The next thing that happens is that if a junior person challenges this senior researcher’s work, the senior guy, because he/she is so sure he/she is right, will go all out to destroy the credibility of the junior researcher, and usually they are successful because they know the right people in the right places. They have no moral qualms in doing this because they know they are right. I have often seen this situation; a senior scientist’s work is challenged in a paper submitted to a journal where the action editor is a friend of the senior guy, and together they torpedo the work of the junior guy. An alternative scenario is that the senior guy gets to publish in his (it’s usually a he I guess, so I will give up my gender neutrality now) friend’s journal, but the junior guy, much worse connected with action editors, get screwed.

      Because of this power asymmetry, it is vital that one teaches researchers to mistrust their own ideas throughout their lives. A little negativity would go a long way, sort of like the opposite of a power pose. Andrew, you should write a book on unpower posing, with a puppet on the cover with its face in its hands, subtitle “Why assuming you are probably wrong will make you rich and famous, or at least make you into a real scientist”.

      • The Researcher should stay skeptical for his own good. But the external ecosystem needs to stay even more vigilant for the greater good. And because the system ought to work *in spite* of an individual researcher that won’t play by the rules.

        Getting some of the things you describe to stop is the better course of action: e.g. Editors publishing based on credentials alone. Junior persons get victimized because *other* senior academics stay passive.

        We are focusing to much on the individual bad players and not enough on the total passivity & failure of the external system that’s supposed to keep them in check. Every time some crappy / flawed work is hyped by NYT etc. why don’t the Psych. Associations respond by issuing a critical Press Release?

        • I suspect that the external ecosystem won’t change, simply because there are too many moving parts. My approach is at least doable on a local scale, it proceeds bottom-up: we’ve brainwashed new researchers, asking them to defend defend defend their theories. We could brainwash them to suspect suspect suspect their theories. That may also affect the ecosystem eventually. Right now, because everyone accepts the premise that defending one’s theory is the right way to go, they are reluctant to encourage dissenting views.

        • “why don’t the Psych. Associations respond by issuing a critical Press Release?”

          Because it would be uncollegial. I would be unwilling to write a press release of that sort when it’s a friend of mine that is under fire. Academics, like soldiers, protect their own from incoming fire.

        • Well, in that case we deserve the crappy studies we get.

          Perhaps knowing that a “friend” wrote the crappy study will soothe our indignation.

        • “The Researcher should stay skeptical for his own good. But the external ecosystem needs to stay even more vigilant”

          The individual researcher needs skepticism both for his/her own good and for the good of the system — the system will only change if enough individuals change.

          “why don’t the Psych. Associations respond by issuing a critical Press Release?”

          Because most of the Psych Associations are in the business of promoting their profession, not in the business of intellectual honesty.

        • Martha:

          Yeah, just for example, I used to link on the blogroll to the British Psychological Society Research Digest, but after awhile I got tired of how they’d hype everything. It was my impression that the editor of the site was trying his best but he was too compromised by his general mission which was to promote the field of psychology.

          In some ways I think econ and poli sci are in better shape, because each of these fields has internal divisions, hence there’s less of a goal in either case to promote the field as a whole.

        • Are you singling out the Psych Association?

          Or is this bit about intellectual honesty being irrelevant a general sentiment about all Academic Associations.

        • I think that most (if not all) professional associations have as one of their goals the promotion of their profession, but that the degree of intellectual honesty varies from profession to profession. My impression is that in general, it is higher in the physical and mathematical sciences than in the social sciences, but is especially low (in my impression) in psychology. (See the Ioannidis references a few posts upthread for some idea of why I believe this.)

        • > higher in the physical and mathematical
          Well in math, replication is low cost and repeatedly taken on by other qualified mathematicians – just requires their time and pen and paper to redo the steps of the proof (maybe computational resources these days).

          Suspect physical science is similar – sloppiness is likely to be noticed and caught fairly quickly.

          Actually, I think Ioannidis had good timing in that we he started to look at replication there were a number of gene expression studies being published that other groups had data in hand to see if what was claimed in the publications would replicate. With clinical trails, it took years to do a similar study and there always seemed to differences that could explain the non-replication.

        • @Rahul

          I do think that mostly different opportunities/costs drive this.

          The difference in intellectual integrity would need to be explained by some selection and training mechanism.

        • @Keith (and Rahul):

          Keith seems to be bringing in medical studies, which I didn’t mention in my comparison. My impression is that medical (and to some extent biological in general) practices in intellectual honesty lie somewhere between the mathematical and physical sciences and the social science. Some further comments on this:

          I agree with Keith that replication is easier (especially less costly) in math and in theoretical physics than in experimental sciences.

          In experimental sciences, incentives can some into play. They may be “social,” (such as pressures to conform to a particular “school” of thought), or emotional (e.g. attachment to a theory), or financial (e.g,, in many medical studies, potential profits from a drug or device are involved).

          But there are also differences from field to field. In particular, many psychological experiments are not as complicated or expensive as medical clinical trials. In fact, some require less work than checking many mathematical or theoretical physics results.

          So this gets down to Keith’s mention of selection and training, which do vary from field to field. Focusing on psychology in particular:

          1. My impression is that (on average) statistical training in psychology is poorer than in most fields. Often psychology departments have their own statistics courses, which (in my impression) are often weaker than statistics courses in statistics, engineering, biology, etc.

          2. My impression is that (on average) selection in psychology is at a lower level than in most other fields. My evidence is purely anecdotal, but includes things such as:
          a. At my university, psychology is one of the largest (perhaps the largest) undergraduate major. (My impression from a little web browsing is that it is one of the most popular majors nationwide as well). My impression is that one reason for its popularity is that it is easier than most majors — so gets many students who have not passed the core courses in other majors. Thus the field appears to be starting out with a lower-than-average pool from which to select graduate students. But, of course, a large undergraduate major needs many TA’s, thus many graduate students. I think you get the idea.
          b. A physician once told me that she had chosen “the easy route to medical school”: an undergraduate psychology major. (Indeed, my university’s psychology department offers a BS in psychology; my understanding is that it was started to cater to premed students.)

        • Martha:

          I could be wrong but based on some things I’ve heard, I suspect that medical trials are the worst of all. Not in terms of forking paths of the analysis, but forking paths in measurement (i.e., falsifying results or cheating on subjective measurements) and data exclusion (dropping patients who are doing well in the control group and patients who are doing poorly in the treatment group).

        • Martha:

          With regard to Tamiflu what was most remarkable was that the “whole record” (apparently) was obtained.

          That is very rare and that’s the only case like that I am aware of.

          I am biased about clinical research because that were I worked (both as a researcher and regulator.)

          For instance, as an old colleague pointed out to me earlier this week – if a regulator has the full data set and someone publishes an article that differs substantially from that and the regulator becomes aware it – they are prevented by law from saying anything about the difference.

        • @Andrew:

          For sure, any shenanigans in a medical trial have the worst impact.

          The downside of a side effect not disclosed are far worse than anything the Himmicanes or Cuddy paper will ever cause.

        • @Keith:
          I second Rahul’s request for more information about the perverse law (although I am not horribly surprised that it exists).

        • @Keith
          Thanks for the links. The first one does look like you need to be a lawyer to understand it. The second is eye-opening, but does look like at least a little progress toward transparency.

  6. Years ago I heard someone suggest these three questions for assessing a work of art:

    1. What was the artist attempting to do?
    2. Were they successful?
    3. Was it worth doing?

    I think those apply equally well to assessing research.

    • Chris:

      There are many cases of successful art, and for that matter successful research, that were created by accident, where the artist or researcher was just mucking around, or maybe just trying to do something to pay the bills, and something great came out of it.

      • Andrew,

        I should have elaborated some on the three questions.

        “Why was the artist attempting to do?” is a challenge to the viewer to bring something to the table. It’s not meant to provoke a judgmental response. The idea is for the observer to think about what the creator (artist, scientist, other) had in mind when they created their work. One can look for “Big Picture” intent and/or specific intent behind a particular piece. (There may be no Big Picture intent behind a work created when just mucking around but I think that there’s specific intent behind most work even if it’s just “I discovered something aesthetically appealing to me and I took it as far as it would go.” )

        “Were they successful?” is an opportunity for some semblance of objectivity – at least an attempt at it. Sometimes you think you see what the artist had in mind but they didn’t quite pull it off. Other times you see a composition that nails it.

        “Was it worth doing?” has no pretense of objectivity. You like what you like and don’t what you don’t. For example, you could acknowledge an artist’s or scientist’s technical skill in creating something but not be taken with the result. Conversely, you might encounter a work which is technically flawed but which opens your eyes to something you’d never considered before thereby making it something which was worth doing.

    • The answers in this case:

      A1. Get fame. Advance a career.

      A2. Yes. Pretty much. Worst case, all news is good news.

      A3. Sure, for Cuddy, yes. For the rest, no net gain. We got a fad. That too will pass.

  7. I am reading “How the Body Knows Its Mind”. It seems to have a lot of iffy psychological stuff that may or may not hold up under replication. Sort of annoying.

  8. Power-pose power is why, when, on the one hand, you see pictures of, say, Albert Einstein, and, on the other, “stills” from movies starring, say, Charlton Heston, it’s so hard to tell the one from the other.

Leave a Reply to Curious Cancel reply

Your email address will not be published. Required fields are marked *