Skip to content
 

The myth of the myth of the myth of the hot hand

Phil pointed me to this paper so I thought I probably better repeat what I wrote a couple years ago:

1. The effects are certainly not zero. We are not machines, and anything that can affect our expectations (for example, our success in previous tries) should affect our performance.

2. The effects I’ve seen are small, on the order of 2 percentage points (for example, the probability of a success in some sports task might be 45% if you’re “hot” and 43% otherwise).

3. There’s a huge amount of variation, not just between but also among players. Sometimes if you succeed you will stay relaxed and focused, other times you can succeed and get overconfidence.

4. Whatever the latest results on particular sports, I can’t see anyone overturning the basic finding of Gilovich, Vallone, and Tversky that players and spectators alike will perceive the hot hand even when it does not exist and dramatically overestimate the magnitude and consistency of any hot-hand phenomenon that does exist.

In summary, this is yet another problem where much is lost by going down the standard route of null hypothesis testing. Better, in my view, to start with the admission of variation in the effect and go from there.

I also used this as an example in the last five paragraphs of section 3 in this paper.

21 Comments

  1. Steve says:

    I think the persistence this paper is picking up is even more basic than the scenarios you describe.

    They don’t control for whether a player is at home and people just shoot better at how so that would explain the persistence in beating expectations of 1-2%. They also don’t control for match quality–some guys just don’t defend picks well and some team run picks well so you’ll see players persistently beat expectations if they keep exploring the same weakness.

  2. Nony says:

    One myth I’d like checked into is “clutchness”. I often hear that some players (or coaches) will get you to the playoffs, but not win the big one (and sometimes the converse). Wonder how much this is making too much soup from one onion.

    Also, as a sports fan, I just think it’s better to get into the playoffs, even if squeaking by. You might get luck, like the Giants. ;-) I think losers make perfect the enemy of better and stay bad forever because they are “building for a Super Bowl” instead of just trying for the playoffs..

  3. Steve Sailer says:

    Has anybody studied the “cold hand?”

    • Andrew says:

      Steve:

      I mention the cold hand in the paper linked to at the end of the post:

      There is little debate that a ‘cold hand’ can exist: it is no surprise that a player will be less successful if he or she is sick, or injured, or playing against excellent defense. Occasional periods of poor performance will manifest themselves as a small positive time correlation when data are aggregated.

      But i have not seen any studies of this. Indeed, the persistent framing of correlations in terms of the “hot hand” rather than the “cold hand” seems like an example of the one-way street fallacy.

      • Steve Sailer says:

        So, if you add up all the causes of Cold Hands, how much does their absence explain Hot Hands?

      • Steve Sailer says:

        I don’t watch a lot of basketball anymore, but I did see the peak game of Linsanity (the most famous recent example of an extended Hot Hand) when Jeremy Lin scored 38 on the defending champion Lakers. There seemed to be three main things going on in that game:

        A. Lin is an NBA-worthy talent who had gone overlooked, so nobody on the Lakers had devoted much study to how to stop Lin

        B. The Lakers’ Derek Fisher was way too old to guard a young, athletic guy like Lin

        C. And on those occasions when the Lakers did succeed in getting somebody close to Lin, he made an improbable number of difficult shots. I figured at the time, he’d regress toward a lower mean as the shots stopped falling. And that happened.

  4. Tim DeRoche says:

    Haven’t studied it. But experienced it many times.

    • Phil says:

      You’re joking, perhaps? I don’t think anyone could be sure that they’re in the middle of a “hot streak” when that means a 2-percentage-point increase in shooting percentage after controlling for a bunch of other variables.

      _Feeling_ like you’re hot, though…sure, we’ve all been there.

  5. Thomas says:

    This reminded me a of discussion I had many years ago at the philosophy blog Certain Doubts. Michael Wheeler suggested the example of Ichiro Suzuki, who thinks he will get a hit even though his batting average (like all even very good batting averages) says the odds are against him. He believes that he’ll get a hit, and the question at the time was whether he is committed to the claim that he knows he will. I argued that we should not say that his belief is based only on his batting average, but on much more “local” considerations. So he believes he’ll hit the ball and he thinks he has good reasons to believe it it. He can (and should) claim he knows he will, even though he’ll also (at some level of abstraction, or in some moods) admit that he’s probably wrong about 2/3 of the time. If he’s honest, however, he’ll probably also say there are times when he doesn’t really believe he’ll hit the ball, and in facts swings and misses. Without knowing it, I think I was attributing a belief in the “hot hand” to him.

    • Steve Sailer says:

      That recalls the observation of a lesser PGA golfer about the great Arnold Palmer: Arnie believes he _deserves_ to make every 20-foot putt.

      • Phil says:

        John Updike said something similar about Ted Williams: “This man, you realized—and here, perhaps, was the difference, greater than the difference in gifts—really intended to hit the ball.”

        On a completely different subject, reading that sentence prompts me to reflect on my own writing. I have a tendency towards parenthetical notes, to excess I believe, so when I find myself writing something like Updike’s sentence I almost always rewrite it; in this case, I would write “This man, you realized, really intended to hit the ball — and here, perhaps was the difference, greater than the difference in gifts.” The advantage of Updike’s formulation here is suspense: he’s about to let us in on what made Williams special, and he wants us to know it. But his sentence is very graceless. Better still, perhaps, would be something like “Seeing him at the plate, you realized there was a difference between Williams and the other ballplayers even bigger than the difference in gifts: this man really intended to hit the ball.”

  6. K? O'Rourke says:

    Bring’s to mind my old policy prof’s comment of regret a year or so ago that new business policy profs no longer consult on policy with organizations but rather just bring their deep statistical insight to bear on readily available observational data sets and draw interesting conclusions [most likely about presence/absence of constant effects] …

    Your paper http://www.stat.columbia.edu/~gelman/research/published/bayes_management.pdf might help make the pointlessness of that more clear.

  7. Dan says:

    Hi Andrew – I’m a fan of your blog and work – probably guilty of some of your criticisms of my field (econ), hopefully not all.
    Agree with most of your points except 2. My paper here shows the empirical results in this area (including the new paper) is consistent with existence of much larger effects, eg 40% shooters sometimes becoming 60-70% shooters
    http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2012.676467#.UyBp6Y3N81A
    wp version, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1805843
    I think you allude to something similar yourself when saying much is lost with standard null hypothesis testing.
    Re your 4th pt – while I agree that hot hand bias is well established, many/most seem to have interpreted GVT to show there is no hot hand at all.
    Here is a piece Jeremy Arkes and I wrote recently summing up our thoughts on the subject:
    http://www.psmag.com/navigation/books-and-culture/stop-denying-hot-hand-basketball-streak-75519/

    • Andrew says:

      Dan:

      The paper discussed above says, “Our estimates of the Hot Hand effect range from 1.2 to 2.4 percentage points in increased likelihood of making a shot,” and that seemed consistent with the effects in the 2% range that I’ve seen before. An effect of 25 percentage points is a lot more, of course. If there’s an average effect of 2 percentage points but it is sometimes as high as 25 percentage points (and, I assume, sometimes as low as -21 percentage points), then indeed it would be important for players and coaches to identify when they’re in the “+big” setting, when they’re in the “-big” setting, and when not much is going on (as seems to be the case on average).

      Also, thanks for the link to your PSmag article. I like this passage:

      The “hot hand bias”—the tendency to impulsively infer a player is hot, based on limited data—is still alive and well. The behavioral researchers were correct to identify this as an important cognitive error. But this does not mean there is no hot hand at all. A player who hits a few tough shots in a row may indeed be the best option for the team’s next shot.

      But, again, if the average difference is 2% and it’s sometimes as high as +25%, it stands to reason that there have got to be some big negative differences (that is, cases where the probability of making a shot is 20% or so lower after just making a shot). So maybe you should add a sentence to the end of the above paragraph, something like:

      And, in other settings, a player who hits a tough shot may be a surprisingly poor choice for the team’s next shot. Part of a coach’s job is to guess (with accuracy better than chance) when there is streakiness and when there’s not.

      • Dan says:

        Thanks very much for your reply Andrew. Great point re there being big negative differences after recent made shot(s) as well. Your suggestion for the sentence to add is a good one – but too late to revise that piece I’m afraid. But will keep this in mind in future discussion.

        Also should say I especially like your pt 1, how we are not robots so it is clear some effect exists. What’s surprising/interesting is that most behavioral researchers (Kahneman etc) do *not* acknowledge this point. Hence, all the confusion/controversy etc

        Best – Dan

  8. Nony says:

    SJ Simon (Why You Lose at Bridge) recommends to ride your luck (some) and to call it a night when you are having a hard time. I’m sure he understood the statistical fallacy, but his main thing was to consider the psychological factor. The issue of desperation affecting judgment.

    • Nony says:

      And even if there’s no net effect on your $$ winning from bridge (people played for money back then), I think his advice would improve one’s “happiness factor”. ;-)

      Although, we do need to think about the gambling compulsion factor (I believe it is part of the psychology of game playing and sports, irrespective of money on the line) and how it works on us, etc. Sometimes, a good solitary workout and some sunshine is better than the stress of battle.

  9. […] Study Says The myth of the myth of the myth of the hot hand – Andrew Gelman […]

  10. Steve Sailer says:

    It would seem like in basketball, there are two separate issues: free throw shooting and field goal shooting. The latter is more glamorous, but there are far more issues involved in analyzing it (e.g., defensive adjustments).

    The problem with free throws, though, is that there is an upper ceiling (100%) that most NBA players are pretty close to already (e.g., the league average is what, around 75%).

    Thus, it might be worth studying bad free throw shooters like Wilt Chamberlain or Shaq to see how often they got hot hands and cold hands. Chamberlain’s free throw shooting season averages are all over the place from a fair to middling .613 to an abysmal .380.

    Chamberlain was a complex free throw shooting machine that could easily go wrong: he had long arms, not particularly deft hands, and an extremely complicated psyche (he bored easily, wasn’t all that focused upon winning, and was acutely aware that he didn’t particularly deserve to his natural gifts). So, I’d say that his most famous hot hand (making 28 of 32 free throws in his 100 point game) was more a moment when his multitudinous reasons for having a cold hand happened to be missing.

  11. Thomas says:

    If there’s a sort of hot hand, what about a cold hand? A player who has a string of luck so bad that they can’t score a single point, despite multiple trials and an average record that would defy this eventuality?

Leave a Reply