You won’t be able to stop staring at this original Hot Hand preprint

To continue with our basketball theme, here’s the preprint of the original hot hand paper, “Misperception of Chance Processes in Basketball,” by Amos Tversky, Robert Vallone, and Thomas Gilovich, from 1985 or so. I remember when it was floating around and everybody was talking about it. When discussing the hot hand with Josh Miller the other day, I remembered I had this preprint in my filing cabinet.

Screen Shot 2015-07-16 at 2.16.23 AM

Here it is. Cool, huh? Even if they did make a mistake in their estimation and then, thirty years later, doubled down and tried to minimize the extent of their error.

Josh Miller’s thoughts

Josh writes:

Question: who coined “hot hand fallacy”?

GVT call it a cognitive illusion in their 1985 paper, and their 2 follow up papers in Chance on 1989.

If you notice in their 1989 “Cold Facts” Chance paper, they are careful to say their result applies exclusively to basketball, because that is the domain in which they asked about beliefs. On the one hand, this makes sense, but really? Imagine finding momentum in human performance in all other sports except basketball, what do you do say then?

While basketball season is just beginning, baseball is in its final throes. Don’t forget Stephen J. Gould had something to say after reading GVT, and also wrote something up for chance in 1989.

Here is Gould’s quote: “Everybody knows about hot hands. The only problem is that no such phenomenon exists. Stanford psychologist Amos Tversky studied…”

He goes on with an attempt to grant Joe DiMaggio an exception, but gives up on it in the end.

Here is Larry Summers, in 2013, in a similar spirit, admonishing the Harvard Basketball team for believing in the hot hand: http://www.nytimes.com/2013/05/05/magazine/larry-summers-and-glenn-hubbard-square-off-on-our-economic-future.html?pagewanted=all

Summers was probably just trying to connect with them, but he has cited GVT’s paper (link here if you want: http://www.jstor.org/stable/2353070?seq=1#references_tab_contents )

Hopefully things will shift, and people don’t have to be embarrassed anymore when talking about the hot hand (though they *sometimes* should be!).

Perhaps people will be more interested in conducting research on momentum in human performance, and not just looking at streak patterns? What a thought! A streak is a data pattern, but the “Hot hand” is a concept, it is a temporary elevation in ability/talent/prob. of success, it should have a mechanism(s), and there must be other things to measure besides, as you call it, “weak” binary data. These two things were entirely confounded in the original GVT study, because the focus was on making light of athletes who said things like: “the basket seems to be so wide. No matter what you do, you know the ball is going to go in” (Purvis Short). Now it may be intellectually respectable to take a look.

Interesting. I like the idea of studying performance more directly rather than obsessing over the data patterns.

15 thoughts on “You won’t be able to stop staring at this original Hot Hand preprint

  1. “Larry Summers, in 2013, in a similar spirit, admonishing the Harvard Basketball team for believing in the hot hand ….Now it may be intellectually respectable to take a look.”

    There is no surer, quicker, or more permanent way to destroy a field than to introduce statistics as the main tool of inquiry. Thank god Newton didn’t have p-values.

    • Mark:

      Monkey Cage has been using Buzzfeed-style headlines for awhile. But my co-bloggers are better at it than I am. Their headlines really seem like things that people might want to click on, whereas my headlines just read like Buzzfeed parody. It’s surprisingly difficult for me to pull it off convincingly.

  2. What is all this talk of the hot hand fallacy being refuted. This seems to have been for a long time primarily summarized as a discussion of their now being a significant hot hand… statistically. But the newfound effects are still small relative to the beliefs people have about them are they not? I don’t think that analysis has been done but I’m pretty sure watching basketball fans that their belief in the effect doesn’t amount to a shift in probability of hitting a shot of a few percentage points. On the contrary, it borders on near certainty.

    • Dear Psyoskeptic

      There are two points of discussion: (A) the hot hand, and (B) hot hand beliefs.

      (A) The hot hand.

      In each controlled data set (and the semi-controlled Three Point Contest) the point estimate of the *mean* effect across players is large in basketball terms, and large enough to be significant statistically. Yes, there is some uncertainty in the effect size, but the large point estimate in GVT’s study replicates as meaningfully large in the NBA’s Three Point contest, and our Spain study.
      There is something else that is reassuring; we know that the estimation procedure is very conservative and, in expectation, will produce underestimates of the hot hand effect. Why? Attenuation in effect from two sources:

      (1) Measurement error: we are trying the estimate how much a player’s ability/talent/prob. of success rises in the hot state. We don’t know when a player is hot/in rhythm/in the zone. We use a crude data pattern as a proxy, a streak. This is a weak signal, and we are likely mixing many “states” together. If you are interested and want to see how severe the underestimation of the hot hand effect can be under very reasonable assumptions for parameters in hot hand models, you can check the appendix of our 2014 “Cold Shower” paper here: http://ssrn.com/abstract=2450479, or check our Daniel Stone’s treatment when looking at serial correlation here: http://ssrn.com/abstract=1805843, or Jeremy Arkes’ treatment here: http://jse.sagepub.com/content/14/4/401.abstract

      (2) heterogeneity in effect: we shouldn’t expect every player to have the same shift in probability of success in the hot state, some perhaps none at all; the conventional wisdom is that some players have the tendency to get hot, and other do not. In our 2014 Study we have some interesting evidence of this. So, if this is true, then when you estimate an *average* effect, you are mixing these tendencies together, and diluting the effect on the “hot hands” or “streaky” players. If you are interested in estimating how big the hot hand can be, you aren’t going to find it this way. You need to admit there is much you cannot know about how big the hot hand can be with this approach. Fortunately, in our 2014 we have enough data from individuals to see that the effect can be quite large for certain individuals. To really focus the mind, with an anecdote, take Craig Hodges, who shot around 56% in the NBA’s Three-Point Contest over many years (around 450 + shots). His longest streak of misses was 5, which is unusually low for 0.56 success probability. What was his longest streak of hits? 19.

      So, should the current evidence lead us to infer that the hot hand effect is small? Absolutely not.

      (2) Hot hand beliefs

      There are spectator beliefs (fans & announcers) and decision maker beliefs (players & coaches), and it is useful to make a distinction here.

      Decision Maker Beliefs:
      If you read the original GVT paper, the decision maker beliefs (76ers surveys) look pretty good in light of what we know now. No need to make fun of Purvis Short, and others. Take a look.

      Further, in our 2014 paper, we find that teammates are pretty good at separating between each other in terms of their *tendency* to become hot. Is this evidence that they can identify in real time when a teammate is hot? No. Is anyone good or bad at identifying the hot hand in real time? Based on the current evidence, this is pure speculation.

      Spectator Beliefs:
      We would expect spectator beliefs to be worse. First they aren’t as experienced, and second their beliefs aren’t disciplined by the need to make decisions based on them. Finally, there is the question, how do you measure beliefs and how do you interpret their responses? This is non-trivial. When you see fans and announcers get excited, well it is more fun to talk about the hot hand. What would they do if you made them put their money where their mouth is? We haven’t seen any one test beliefs this way. But, if you look at the numerical reports in fan surveys conducted by GVT, yes, fans don’t look so good. Are we surprised by this? Would we have been surprised by this in 1985?

      Summary

      The 1985 paper was highly original in its design. They identified what seemed to be the perfect environment for testing if the biased beliefs in sequential judgement and decision making that had been identified in the lab could be extended to a field environment in which mistaken beliefs were potentially costly. The idea itself is the achievement, the rest should be just the application of standard methods. My former colleague Martin Dufwenberg (now at U. of Arizona) has proposed that journals should publish exactly on this criteria (see here: http://www.u.arizona.edu/~martind1/Papers-Documents/krhtcfses.pdf).

      We cannot forget that for the 1985 GVT paper, the main result and surprising take away point for the general interest scientific community, and the lay public, was that the belief in the hot hand was a massive and widespread cognitive illusion. This was counter to anyone’s intuition who has ever played basketball (or any other sport for that matter). For a psychologists it was surprising too: it is known that the human capacity for attention and effort varies and can greatly affect performance; the same can be said for confidence, not to mention other physiological factors that influence shooting ability. For an economist it was surprising that expert practitioners would insist on holding and acting upon inaccurate beliefs, and refuse to change them in the face of evidence and incentives to do so.

      The evidence no longer supports this sensational result.

Leave a Reply to Andrew Cancel reply

Your email address will not be published. Required fields are marked *