Someone points me to this post by Scott Alexander, criticizing the work of psychology researcher Carol Dweck. Alexander looks carefully at an article, “Mindset Interventions Are A Scalable Treatment For Academic Underachievement,” by David Paunesku, Gregory Walton, Carissa Romero, Eric Smith, David Yeager, and Carol Dweck, and he finds the following:
Among ordinary students, the effect on the growth mindset group was completely indistinguishable from zero, and in fact they did nonsignificantly worse than the control group. This was the most basic test they performed, and it should have been the headline of the study. The study should have been titled “Growth Mindset Intervention Totally Fails To Affect GPA In Any Way”.
As Alexander reports, the authors “went to subgroup analysis.” But we all know the problems there. Garden of forking paths, anyone? Again, let me emphasize that I don’t think that preregistration is the best solution to the garden of forking paths; rather, I recommend multilevel modeling, looking at all interactions that might be of interest, not just pulling out a few dramatic comparisons.
At this point I’m in a cloud of cognitive dissonance. On one hand, I met Dweck once and she seemed very reasonable. On the other hand, Alexander’s criticisms do seem reasonable, and it doesn’t help that the article in question was published in . . . yup, you guessed it, Psychological Science.
So really I don’t know what to think.
But what really amazed me were two things:
1. I’d never heard of this guy and his blog has about a zillion comments. There clearly are large corners of the internet that I didn’t know about.
2. It was also striking that 100% of the commenters thought the study in question is B.S. I have no idea, but Dweck is a respected resesarcher. I don’t think she’s in Daryl Bem or Ellen Langer territory.
The person who sent the original message replied to me:
There definitely been complaints from some corners about Dweck’s work not replicating, but also lots of followers doing other mindset experiments in her tradition.
Re that, these are the two posts preceding the analysis of that study:
The blog posts moderately often about bad stats, and Vox and the Atlantic link to him occasionally. Some other random stats related posts:
More alcoholism treatment and the false positive psychology paper:
The claim false rape accusations are less than 1/30th as common as being struck by lighting:
The “perceptions of required ability by field” study:
Hmmm, I guess I should look into this in more detail. Maybe I’ll talk with some of my psychology colleagues. In any case, I’m still impressed by Alexander getting hundreds of comments on that post—he must be doing something right to be getting this sort of attention and careful reading!
P.S. More here.
P.P.S. The person who sent the above message informs me that an author of the paper said that they have had another successful replication since, and will be preregistering their next one on Open Science Framework. If their effect is real and works in the preregistered many-school replications then it will generate a huge amount of social value by helping millions of kids in school.
P.P.P.S. David Yeager responds in some comments below. I appreciate his active engagement in post-publication review, which should be a model for researchers everywhere.