J. Michael Steele explains why he doesn’t like the above saying (which, as he says, is attributed to statistician George Box). Steele writes, “Whenever you hear this phrase, there is a good chance that you are about to be sold a bill of goods.”
He considers a street map of Philadelphia as an example of a model:
If I say that a map is wrong, it means that a building is misnamed, or the direction of a one-way street is mislabeled. I never expected my map to recreate all of physical reality, and I only feel ripped off if my map does not correctly answer the questions that it claims to answer. My maps of Philadelphia are useful. Moreover, except for a few that are out-of-date, they are not wrong.
Actually, my guess is that his maps are wrong, in that there probably are a couple of streets that are mislabeled in some way. Street maps are updated occasionally (even every year), but streets get changed, and not every change is captured in an update. I expect there are a few places where Steele’s map has mistakes. (But I doubt it’s like those old tourist street maps of Soviet cities which, I’ve been told, had lots of intentional errors to make it harder for people to actually find their way around too well.) In any case, I take his general point, which is that a street map could be exactly correct, to the resolution of the map.
Statistical models of the sort that I typically use are different in being generative: that is, they are stochastic prescriptions for creating data. As such, they can typically never be proven wrong (except in special cases, for example a binary regression model can’t produce a data value of 0.6). The saying, “all models are wrong,” is helpful because it is not completely obvious, since it can’t always be proved in special cases.
Recall the saying that a chi-squared test is a “measure of sample size.” With a small sample size, you won’t be able to reject even a silly model, and with a huge sample size, you’ll be able to reject any statistical model you might possibly want to use (at least in the social and environmental sciences, where I do most of my work). This is a simple point, and I can see how Steele can be irritated by people making a big point about it . . . .
But, the trouble is, many people don’t realize that all models are wrong. They want to make statements such as, The probability is 0.74 that the logistic regression model with predictors A,B,and D is correct. This is not the sort of statement I ever want to say.
The point of posterior predictive checking (see chapter 6 of Bayesian Data Analysis, or chapter 8 in our regression book for a less explicitly Bayesian treatment) is to use numerical and graphical summaries to understand what aspects of the data are captured by the model and what aspects are not. The goal is not to check whether the model is “wrong”–after all, all models are wrong–but to see how well it fits. I agree with Steele that external validation is good too.