I’m a big fan of Bill James, but I think he might be picking up the wrong end of the stick here.
The great baseball analyst writes about what he calls the Law of Competitive Balance. His starting point is that teams that are behind are motivated to work harder to have a chance of winning, which moves them to switch to high-variance strategies such as long passes in football (more likely to score a touchdown, also more likely to get intercepted), etc. Here’s Bill James:
Why was there an increased chance of a touchdown being thrown?
Because the team was behind.
Because the team was behind, they had an increased NEED to score.
Because they had an increased need to score points, they scored more points.
That is one of three key drivers of The Law of Competitive Balance: that success increases when there is an increased need for success. This applies not merely in sports, but in every area of life. But in the sports arena, it implies that the sports universe is asymmetrical. . . .
Because this is true, the team which has the larger need is more willing to take chances, thus more likely to score points. The team which is ahead gets conservative, predictable, limited. This moves the odds. The team which, based on their position, would have a 90% chance to win doesn’t actually have a 90% chance to win. They may have an 80% chance to win; they may have an 88% chance to win, they may have an 89.9% chance to win, but not 90.
I think he’s mixing a correct point here with an incorrect point.
James’s true statement is that, as he puts it, “there is an imbalance in the motivation of the two teams, an imbalance in their willingness to take risks.” The team that’s behind is motivated to switch to strategies that increase the variance of the score differential, even at the expense of lowering its expected score differential. Meanwhile, the team that’s ahead is motivated to switch to strategies that decrease the variance of the score differential, even at the expense of lowering its expected score differential. In basketball, it can be as simple as the team that’s behind pushing up the pace and the team that’s ahead slowing things down. The team that’s trailing is trying to have some chance of catching up—their goal is to win, not to lose by a smaller margin; conversely, the team that’s in the lead is trying to minimize the chance of the score differential going to zero, not to run up the score. As James says, these patterns are averages and won’t occur from play to play. Even if you’re behind by 10 in a basketball game with 3 minutes to play, you’ll still take the open layup rather than force the 3-pointer, and even if you’re ahead by 10, you’ll still take the open shot with 20 seconds left on the shot clock rather than purely trying to eat up time. But on average the logic of the game leads to different strategies for the leading and trailing teams, and that will have consequences on the scoreboard.
James’s mistake is to think, when this comes to probability of winning, that this dynamic on balance favors the team that’s behind. When strategies are flexible, the team that’s behind does not necessarily increase its probability of winning relative to what that probability would be if team strategies were constant. Yes, the team that’s behind will use strategies to increase the probability of winning, but the team that’s ahead will alter its strategy too. Speeding up the pace of play should, on average, increase the probability of winning for the trailing team (for example, increasing the probability from, I dunno, 10% to 15%), but meanwhile the team that’s ahead is slowing down the pace of play, which should send that probability back down. On net, will this favor the leading team or the trailing team when it comes to win probability? It will depend on the game situation. In some settings (for example, a football game where the team that’s ahead has the ball on first down with a minute left), it will favor the team that’s ahead. In other settings it will go the other way.
James continues:
That is one of the three key drivers of the Law of Competitive Balance. The others, of course, are adjustments and effort. When you’re losing, it is easier to see what you are doing wrong. Of course a good coach can recognize flaws in their plan of attack even when they are winning, but when you’re losing, they beat you over the head.
I don’t know about that. As a coach myself, I could just as well argue the opposite point, as follows. When you’re winning, you can see what works while having the freedom to experiment and adapt to fix what doesn’t work. But when you’re losing, it can be hard to know where to start or have a sense of what to do to improve.
Later on in his post, James mentions that, when you’re winning, part of that will be due to situational factors that won’t necessarily repeat. The quick way to say that is that, when you’re winning, part of your success is likely to be from “luck”; a formulation that I’m OK with as long as we take this term generally enough to refer to factors that don’t necessarily repeat, such as pitcher/batter matchups, to take an example from James’s post.
But James doesn’t integrate this insight into his understanding of the law of competitive balance. Instead, he writes:
If a baseball team is 20 games over .500 one year, they tend to be 10 games over the next. If a team is 20 games under .500 one year, they tend to be 10 games under the next year. If a team improves by 20 games in one year (even from 61-101 to 81-81) they tend to fall back by 10 games the next season. If they DECLINE by 10 games in a year, they tend to improve by 5 games the next season.
I began to notice similar patterns all over the map. If a batter hits .250 one year and .300 the next, he tends to hit about .275 the third year. Although I have not demonstrated that similar things happen in other sports, I have no doubt that they do. I began to wonder if this was actually the same thing happening, but in a different guise. You get behind, you make adjustments. You lose 100 games, you make adjustments. You get busy. You work harder. You take more chances. You win 100 games, you relax. You stand pat.
James’s description of the data is fine; his mistake is to attribute these changes to teams “making adjustments” or “standing pat.” That could be, but it could also be that teams that win 100 games “push harder” and that teams that lose 100 games “give up.” The real point is statistical, which is that this sort of “regression to the mean” will happen without any such adjustment effects, just from “luck” or “random variation” or varying situational factors.
Here’s a famous example from Tversky and Kahneman (1973):
The instructors in a flight school adopted a policy of consistent positive reinforcement recommended by psychologists. They verbally reinforced each successful execution of a flight maneuver. After some experience with this training approach, the instructors claimed that contrary to psychological doctrine, high praise for good execution of complex maneuvers typically results in a decrement of performance on the next try.
Actually, though:
Regression is inevitable in flight maneuvers because performance is not perfectly reliable and progress between successive maneuvers is slow. Hence, pilots who did exceptionally well on one trial are likely to deteriorate on the next, regardless of the instructors’ reaction to the initial success. The experienced flight instructors actually discovered the regression but attributed it to the detrimental effect of positive reinforcement.
“Performance is not perfectly reliable and progress between successive maneuvers is slow”: That describes pro sports!
As we write in Regression and Other Stories, the point here is that a quantitative understanding of prediction clarifies a fundamental qualitative confusion about variation and causality. From purely mathematical considerations, it is expected that the best pilots will decline, relative to the others, while the worst will improve in their rankings, in the same way that we expect daughters of tall mothers to be, on average, tall but not quite as tall as their mothers, and so on.
I was surprised to see Bill James make this mistake. All the years I’ve read him writing about the law of competitive balance and the plexiglass principle, I always assumed that he’d understood this as an inevitable statistical consequence of variation without needing to try to attribute it to poorly-performing teams trying harder etc.
How did he get this one so wrong? Here are 6 reasons.
This raises a new question, which is how could such a savvy analyst make such a basic mistake? I have six answers:
1. Multiplicity. Statistics is hard, and if you do enough statistics, you’ll eventually make some mistakes. I make mistakes too! It just happens that this “regression to the mean” fallacy is a mistake that James made.
2. It’s a basic mistake and an important mistake, but it’s not a trivial mistake. Regression to the mean is a notoriously difficult topic to teach (you can cruise over to chapter 6 of our book and see how we do; maybe not so great!).
3. Statistics textbooks, including my own, are full of boring details, so I can see that, whether or not Bill James has read any such books, he wouldn’t get so much out of them.
4. In his attribution of regression to the mean, James is making an error of causal reasoning and a modeling error, but it’s not a prediction error. The law of competitive balance and the plexiglass principle give valid predictions, and they represent insights that were not widely available in baseball (and many other fields) before James came along. Conceptual errors aside, James was still moving the ball forward, as it were. When he goes beyond prediction in his post, for example making strategy recommendations, I’m doubtful, but I’m guessing that the main influence on readers of his “law of competitive balance” is to the predictive part.
5. Hero worship. The man is a living legend. That’s great—he deserves all his fame—but the drawback is that maybe it’s a bit too easy for him to fall for his own hype and not question himself or fully hear criticism. We’ve seen the same thing happen with baseball and political analyst Nate Silver, who continues to do excellent work but sometimes can’t seem to digest feedback from outsiders.
6. Related to point 5 is that James made his breakthroughs by fighting the establishment. For many decades he’s been saying outrageous things and standing up for his outrageous claims even when they’ve been opposed by experts in the field. So he keeps doing it, which in some ways is great but can also lead him astray, by trusting his intuitions too much and not leaving himself more open for feedback.
I guess we could say that, in sabermetrics, James was on a winning streak for a long time so he relaxes. He stands pat. He has less motivation to see what’s going wrong.
P.S. Again, I’m a big fan of Bill James. It’s interesting when smart people make mistakes. When dumb people make mistakes, that’s boring. When someone who’s thought so much about statistics makes such a basic statistical error, that’s interesting to me. And, as noted in item 4 above, I can see how James could have this misconception for decades without it having much negative effect on his work.
P.P.S. Just to clarify: Bill James made two statements. The first was predictive and correct; the second was causal and misinformed.
His first, correct statement is that there is “regression to the mean” or “competitive balance” or “plexiglas” or whatever you want to call it: players or teams that do well in time 1 tend to decline in time 2, and players or teams that do poorly in time 2 tend to improve in time 2. This statement, or principle, is correct, and it can be understood as a general mathematical or statistical pattern that arises when correlations are less than 100%. This pattern is not always true—it depends on the joint distribution of the before and after measurements (see here) but it is typically the case.
His second, misinformed statement is that this is caused by players or teams that are behind at time 1 being more innovative or trying harder and players or teams that are ahead at time 2 being complacent or standing pat. This statement is misinformed because the descriptive phenomenon of regression-to-the-mean or competitive-balance or plexiglas will happen even in the absence of any behavioral changes. And, as discussed in the above post, behavioral changes can go in either direction; there’s no good reason to think that, when both teams perform strategic adjustments, that these adjustments on net will benefit the team that’s behind.
This is all difficult because it is natural to observe the first, correct predictive statement and from this to mistakenly infer the second, misinformed causal statement. Indeed this inference is such a common error that it is a major topic in statistics and is typically covered in introductory texts. We devote a whole chapter to it in Regression and Other Stories, and if you’re interested in understanding this I recommend you read chapter 6; the book is freely available online.
For the reasons discussed above, I’m not shocked that Bill James made this error: for his purposes, the predictive observation has been more important than the erroneous causal inference, and he figured this stuff out on his own, without the benefit or hindrance of textbooks, and given his past success as a rebel, I can see how it can be hard for him to accept when outsiders point out a subtle mistake. But, as noted in the P.S. above, when smart people get things wrong, it’s interesting; hence this post.