If you want to know about basketball, who ya gonna trust, a mountain of p-values . . . or that poseur Phil Jackson??

Someone points me with amusement to this published article from 2012:

Beliefs About the “Hot Hand” in Basketball Across the Adult Life Span

Alan Castel, Aimee Drolet Rossi, and Shannon McGillivray
University of California, Los Angeles

Many people believe in streaks. In basketball, belief in the “hot hand” occurs when people think a player is more likely to make a shot if they have made previous shots. However, research has shown that players’ successive shots are independent events. To determine how age would impact belief in the hot hand, we examined this effect across the adult life span. Older adults were more likely to believe in the hot hand, relative to younger and middle-aged adults, suggesting that older adults use heuristics and potentially adaptive processing based on highly accessible information to predict future events.

My correspondent writes: “This paper is funny, I didn’t realize how strongly the psych community bought the null hypothesis.”

To be fair, back in 2012, I didn’t think the hot hand was real either . . .

But what really makes it work is this quote from the first paragraph of the above-linked paper:

Anecdotally, many fans, and even coaches, profess belief in the hot hand. For example, Phil Jackson, one of the most successful coaches in the history of the National Basketball Association (NBA), once said of Kobe Bryant, in Game 5 of the 2010 NBA Finals: “He’s the kind of guy (where) you ride the hot hand, that’s for sure, we were waiting for him to do that . . . . He went out there and found a rhythm.”

I mean, if you want to know about basketball, who ya gonna trust, a mountain of p-values . . . or that poseur Phil Jackson?? Have you seen how bad the Knicks were last year? Zen master, my ass.

20 thoughts on “If you want to know about basketball, who ya gonna trust, a mountain of p-values . . . or that poseur Phil Jackson??

  1. “I didn’t realize how strongly the psych community bought the null hypothesis”

    I have had the depressing thought that maybe the field of social psych is about ten times bigger than it should be because of p-hacking. This paper is a good example. To whit: it would be useful to know if the hot hand phenomenon actually exists (which I think is still being debated) before I can generate much interest in whether older or younger folks tend to believe in it more.

  2. This post makes me wonder again about the question of whether null hypotheses are always so uninteresting. This has been a frequent point of Andrews, that the null hypothesis is just an empty foil against which to pit a directional hypothesis, and rarely reflects useful content. (Tell me if/how I am misrepresenting you.) But in the realm of behavioral economics, I’m not so sure this is true. I agree it is fairly uninteresting whether the hot hand effect is exactly 0, and a lot of the early controversy over the hot hand research was motivated by people’s belief that it seems unlikely that any skill would show exactly zero autocorrelation. (It sure feels like I have good runs and bad runs in my favorite hobbies, online chess and woodwinds).

    But there IS an interesting null hypothesis lurking in here, which is that the error in people’s average assessment of the hot hand should be exactly zero, neither biased toward thinking it’s bigger than it is or smaller than it is. This was really the main point of the original work–that people systematically overestimate the size of the hot hand, for various reasons including my favorite, that people overestimate the reliability of highly unreliable information (which for obvious reasons is hard to underestimate the reliability of). I don’t think the new statistical analyses really changes my mind much on this matter–the reanalysis shows the hot hand exists, but is small, and I suspect people still overestimate it.

    But back to the main point–the null hypothesis of unbiased belief is a fundamental tenet of traditional economics, which means that it is both drawn from a well-established set of theories and is the foundation of many as well. I anticipate that Andrew will say “ok, so now focus on what really matters, how big the bias is and how variable it is”. Well said, Andrew. But I am still thinking that null hypotheses can be interesting, at least in cases like this.

    • RJB:

      Agreed on what is interesting from the psych/behavioral econ perspective, in fact Tversky & Gilovich argue:

      Furthermore, the present analysis of basketball data does not say whether baseball or tennis players, for example, go through hot and cold periods. Our research does not tell us anything general about sports, but it does suggest a generalization about people, namely that they tend to “detect” patterns even where none exist, and to overestimate the degree of clustering in sports events, as in other sequential data.

      The problem is that the original study concluded that the hot hand was a myth, it was a cognitive illusions driven by misperceptions of chance. But the belief in the hot hand is not a cognitive illusion because the hot hand is not a myth. We we have shown that there is a sizeable hot hand in their data, and in prominent replications.

      You could relax what you mean by cognitive illusion, and appeal to GVT’s betting task in which bettors (players) believe they can see the hot hand, yet, as Tversky & Gilovich conclude: “The players were generally unsuccessful in predicting their performance.” You’d think that if players can detect the hot hand, then they could out-perform chance in a betting task? Seems like evidence for the fallacy. Tom Gilovich has argued that this is the most important piece of evidence against the hot hand, and what led him to switch from his original hypothesis that players and fans *overestimate* the prevalence of streak shooting, to the idea that streak shooting simply doesn’t exist. But this conclusion doesn’t hold. We have re-examined the data from the betting task and found the opposite conclusion, player’s shoot significantly (and substantially) better after a bettor bets hit (vs. miss). This means that either: (1) players can detect the hot hand as it occurs, or (2) because they believe in the hot hand, and bet accordingly, they (mechanically) predict well because the hot hand exists in the data. We discuss both possibilities.

      You say: “the reanalysis shows the hot hand exists, but is small, and I suspect people still overestimate it.”

      Actually the hot hand is not small we find point estimates between 6pp and 13pp across studies, the largest being in GVT’s study. This is not small. Imagine a baseball player going from a .300 hitter to a .360 hitter, that’s big.

      You say: “But back to the main point–the null hypothesis of unbiased belief is a fundamental tenet of traditional economics, which means that it is both drawn from a well-established set of theories and is the foundation of many as well”

      agreed, I saw a nice comment on twitter today affirming this view.

      on the other hand, we also care about how biased people’s (operational) beliefs are, because small biases may not need to be corrected if the consequences are slight. Being only slightly different from the null isn’t that interesting (practically speaking), regardless of which side you are on. This is not to say that players don’t over-react to the hot hand, anecdotes abound and players themselves believe that they can over-react, but I am unaware of any evidence that they systematically over-react, and the evidence I have seen on shot allocation decisions seems to suggest they are pretty good at it.

  3. “I didn’t realize how strongly the psych community bought the null hypothesis”

    I think the null hypothesis of approximately no effect is plausible here. I think sinking a basket, or especially a few in a row, could increase the players confidence and willing to take risks and/or aggressiveness, which could influence how and under what circumstances they attempt a subsequent shot, which could influence their shot, and thus the probability of sinking it. I also think that the influence could be to decrease the probability of sinking it, and that the actual effect could vary dramatically across many different combination of other factors (e.g. the player and all his characteristics at the time (fatigue,etc), the opposing team, the defensive players guarding the shooter, how well both teams are generally playing at the time of the shots, etc), and that the overall net effect could be close to nothing.

    I think this problem is similar to the syncing menstrual cycles phenomenon, which was debunked a number of years ago – at least I feel the evidence was strong enough to debunk it. I also think both of these problems, and others are described in a book I read in the last year, Probabilities: the little numbers that rule our lives, by Peter olofsson.

    • Jake:

      right, but your statements refer to percentages, i.e. play-by-play statistics. People’s beliefs about play-by-play data do not match play-by-play data on many dimensions, which is why play-by-play data exists to begin with.

      When people say a player has the hot hand, or is in the zone, or in rhythm, or whatever, they are referring to their feeling that there has been a shift in the player’s underlying probability of success. What cues that feeling? What causes that shift? Probably many things. It seems implausible that controlling for objective shot difficulty (e.g, defense, shot location) that a given player’s probability of success never changes.

  4. I think the most reasonable assumption absent comprehensive evidence is that the hot hand effect may well exist, but is highly variable. To claim, like the authors of that paper, that there is some global truth that holds for all players is naïve in the face of human variability.

    Also, I suspect that if you’re studying this only/mostly in world-class basketball players or other sports pros, you’re operating so close to the performance ceiling that it squashes the effect sizes you get to see. The phenomenon is certainly of broader interest than just pro sports, and you might get much bigger effects in people who haven’t yet exhausted their potential.

    I just happen to have a nice illustration: playing basket ball is my way of exercising regularly, but I’m average at best. I put my free throw hit rate at about 60%. Very very occasionally, I make something like 5 or 7 in a row. The other day, I sank 14 in a row. I know, anecdotal evidence. But it sure felt good!

  5. The fundamental issue is: Do some particular players (e.g. Bryant, Jordan, Bird, Miller, Curry), not necessarily all of them, manifest evidence of having a ‘hot hand’? This issue has NEVER been properly studied scientifically. To do so would require obtaining a context-dependent, player-specific, combination of behavioral, psychological, and psycho-physiological data that have never been acquired yet. Without such, all remains speculation to date. E.g. Maybe Phil J is a Zen master on some but not all occasions; maybe it depends on how much he meditated in the morning on game day. The Truth still remains mysterious right now…

    • Dave:

      The evidence I’ve seen suggests the hot hand definitely exists and the effect could be large. Estimating how it varies by player is more difficult. I agree with you that the way forward is to use better data; there’s a limited amount of information we can get from simply looking at sequences of hits and misses.

      • Andrew, yes exactly, on all scores (pun intended). It would be a great project to proceed further in these directions; could have implications for improving human performance on many fronts, not just hoops. And, my own personal experience has been that ‘The Zone’ does truly exist both physically and mentally; you can even see it on the faces of those who are in it… cf.

        https://m.youtube.com/watch?v=grBmQwLSlDw

    • To do so would require obtaining a context-dependent, player-specific, combination of behavioral, psychological, and psycho-physiological data that have never been acquired yet. Without such, all remains speculation to date.

      I don’t see why you would need all that. All you need to study it scientifically is a model that makes a precise prediction to check against some data. If the “existence of hot-hand” is defined clearly enough you should be able to do it from hit/miss data.

      • Anon:

        I wouldn’t say we’ve reached definitive proof (whatever that would mean) but the evidence from hit/miss data for the existence of hot hand seems pretty compelling to me. Estimating the size of the effect and how it varies, though, that’s more difficult, and I think more granular data will be necessary. The trouble with hit/miss data is that they’re inherently noisy, and it’s hard to get a large enough sample under game conditions. One could collect thousands of hit/miss data from practices, but I think that to make real progress we’d want to know more about the shots. It is possible to gather such data using cameras.

Leave a Reply to Dave Meyer Cancel reply

Your email address will not be published. Required fields are marked *