I happened to be browsing through the Bill James Historical Baseball Abstract the other day and came across this passage on Glenn Hubbard, who he ranks as the 88th best second baseman of all time:
Total Baseball has Glenn Hubbard rated as a better player than Pete Rose, Brooks Robinson, Dale Murphy, Ken Boyer, or Sandy Koufax, a conclusion which is every bit as preposterous as it seems to be at first blush.
To a large extent, this rating is caused by the failure to adjust Hubbard’s fielding statistics for the ground-ball tendency of his pitching staff. Hubbard played second base for teams which had very high numbers of ground balls, as is reflected in their team assists totals. The Braves led the National League in team assists in 1985, 1986, and 1987, and were near the league lead in the other years that Hubbard was a regular. Total Baseball makes no adjustment for this, and thus concludes that Hubbard is reaching scores of baseballs every year that an average second basement would not reach, hence that he has enormous value.
Posterior checking! This would fit in perfectly in chapter 6 of BDA.
This idea is so fundamental to statistics—to science—and yet so many theories of statistics and theories of science have no place for it.
The alternative to the Jamesian, model-checking approach—so close to the Jaynesian approach!—is exemplified by Pete Palmer’s Total Baseball book, mentioned in the above quote. Pete Palmer did a lot of great stuff, and Bill James is a fan of Palmer, but Palmer follows the all-too-common approach of just taking the results from his model and then . . . well, then, what can you do? You use what you’ve got.
What makes Bill James special is that he’s interested in getting it right, and he’s interested in seeing where things went wrong.
A chicken is an egg’s way of making another egg.
To make the analogy explicit: the “egg” is the model and data, and the chicken is the inferences from the model. The chicken is implicit in the egg, but it needs some growing. The inferences are implicit in the model and the data, but it takes some computing.
All the effort that went into Total Baseball was useful for sabermetrics, in part for the direct relevance of the results (a delicious “chicken”) and in part because Total Baseball included so much data and made so many inferences that people such as James could come in and see which of these statements made no sense—and what this revealed about the problems with Palmer’s model.
It’s like Lakatos said in Proofs and Refutations: once you have the external counterexample—an implication that doesn’t make sense—you go find the internal flaw—the assumption in the model that went wrong, often an assumption that was so implicit in the construction of your procedure that you didn’t even realize it was an assumption at all. (Remember the Speed Racer principle?) Or, conversely, if you first find an internal assumption that concerns you, you should follow the thread outward and figure out what are its external consequences: what does it imply that it does not make sense.
P.S. James is doing a posterior check, not a prior check, because his criticism of the Total Baseball model comes from the absurdity of one of its inferences, conditional on the data.