In general, the further individuals, groups, and teams are ahead of their opponents in competition, the more likely they are to win. However, we show that through increasing motivation, being slightly behind can actually increase success. Analysis of over 6,000 collegiate basketball games illustrates that being slightly behind increases a team’s chance of winning. Teams behind by a point at halftime, for example, actually win more often than teams ahead by one. This increase is between 5.5 and 7.7 percentage points . . .
This is an interesting thing to look at, but I think they’re wrong. To explain, I’ll start with their data, which are 6572 NCAA basketball games where the score differential at halftime is within 10 points. Of the subset of these games with one-point gaps at halftime, the team that’s behind won 51.3% of the time. To get a standard error on this, I need to know the number of such games; let me approximate this by 6572/10=657. The s.e. is then .5/sqrt(657)=0.02. So the simple empirical estimate with +/- 1 standard error bounds is [.513 +/- .02], or [.49, .53]. Hardly conclusive evidence!
Given this tiny difference of less than 1 standard error, how could they claim that “being slightly behind increases a team’s chance of winning . . . by between 5.5 and 7.7 percentage points”?? The point estimate looks too large (6.6 percentage points rather than 1.3) and the standard error looks too small.
What went wrong? A clue is provided by this picture:
As some of Wolfers’s commenters pointed out, this graph is slightly misleading because all the data points on the right side are reflected on the left. The real problem, though, is that what Berger and Pope did is to fit a curve to the points on the right half of the graph, extend this curve to 0, and then count that as the effect of being slightly behind.
This is wrong for a couple of reasons.
First, scores are discrete, so even if their curve were correct, it would be misleading to say that being behind increases your chance of winning by 6.6 points. Being behind takes you from a differential of 0 (50% chance of winning, the way they set up the data) to 51% (+/- 2%). Even taking the numbers at face value, you’re talking 1%, not their claimed 5% or more.
Second, their analysis is extremely sensitive to their model. Looking at the picture above–again, focusing on the right half of the graph–I would think it would make more sense to draw the regression line a bit above the point at 1. That would be natural but it doesn’t happen here because (a) their model doesn’t even try to be consistent with the point at 0, and (b) they do some ridiculous overfitting with a 5th-degree polynomial. Don’t even get me started on this sort of thing.
What would I do?
I’d probably start with a plot similar to their graph above, but coding score differential consistently as “home team score minus visiting team score.” Then each data point would represent different games, they could fit a line and see what they get. And I’d fit linear functions (on the logit scale), not 5th-degree polynomials. And I’d get more data! The big issue, though, is that we’re talking about maybe a 1% effect, not a 7% effect, which makes the whole thing a bit less exciting.
P.S. It’s cool that Berger and Pope tried to do this analysis. I also appreciate that they attempted to combine sports data with a psychological experiment, in the spirit of the (justly) celebrated hot-hand paper. I like that they cited Hal Stern. And, even discounting their exaggerated inferences, it’s perhaps interesting that teams up by 1% at halftime don’t do better. This is just what happens when studies get publicized before peer review. Or, to put it another way, the peer review is happening right now! I’ve put enough first-draft mistakes on my own blogs that I can’t hold it against others when they do the same.
P.P.S. Update here.