Some thoughts on the saying, “All models are wrong, but some are useful”

J. Michael Steele explains why he doesn’t like the above saying (which, as he says, is attributed to statistician George Box). Steele writes, “Whenever you hear this phrase, there is a good chance that you are about to be sold a bill of goods.”

He considers a street map of Philadelphia as an example of a model:

If I say that a map is wrong, it means that a building is misnamed, or the direction of a one-way street is mislabeled. I never expected my map to recreate all of physical reality, and I only feel ripped off if my map does not correctly answer the questions that it claims to answer. My maps of Philadelphia are useful. Moreover, except for a few that are out-of-date, they are not wrong.

Actually, my guess is that his maps are wrong, in that there probably are a couple of streets that are mislabeled in some way. Street maps are updated occasionally (even every year), but streets get changed, and not every change is captured in an update. I expect there are a few places where Steele’s map has mistakes. (But I doubt it’s like those old tourist street maps of Soviet cities which, I’ve been told, had lots of intentional errors to make it harder for people to actually find their way around too well.) In any case, I take his general point, which is that a street map could be exactly correct, to the resolution of the map.

Statistical models of the sort that I typically use are different in being generative: that is, they are stochastic prescriptions for creating data. As such, they can typically never be proven wrong (except in special cases, for example a binary regression model can’t produce a data value of 0.6). The saying, “all models are wrong,” is helpful because it is not completely obvious, since it can’t always be proved in special cases.

Recall the saying that a chi-squared test is a “measure of sample size.” With a small sample size, you won’t be able to reject even a silly model, and with a huge sample size, you’ll be able to reject any statistical model you might possibly want to use (at least in the social and environmental sciences, where I do most of my work). This is a simple point, and I can see how Steele can be irritated by people making a big point about it . . . .

But, the trouble is, many people don’t realize that all models are wrong. They want to make statements such as, The probability is 0.74 that the logistic regression model with predictors A,B,and D is correct. This is not the sort of statement I ever want to say.

The point of posterior predictive checking (see chapter 6 of Bayesian Data Analysis, or chapter 8 in our regression book for a less explicitly Bayesian treatment) is to use numerical and graphical summaries to understand what aspects of the data are captured by the model and what aspects are not. The goal is not to check whether the model is “wrong”–after all, all models are wrong–but to see how well it fits. I agree with Steele that external validation is good too.

11 thoughts on “Some thoughts on the saying, “All models are wrong, but some are useful”

  1. I think the analogy to a map is a bad one, in the way he has framed the question. All maps are wrong. The street map does not have elevation, the location of trees in peoples' yards and parks, the locations of storm drains, etc. It is an abstraction that is used to locate streets, and the other information is less important given this goal. Nevertheless, every map leaves something out, so it is not "true".

    In the same way, when I build a model of, say the dynamics of an animal population, I leave out MOST of the variables affecting the population, electing only to include the handful of important covariates that might allow me to predict future states. So, my model is wrong — but its useful.

  2. All actual street maps are wrong for another reason: Maps are copyrighted, and copyright holders build in small, insignificant errors in random places on the map so as to be able to enforce their copyright in court. A copier won't know where the errors are and will reproduce them, but a map legitimately produced will be very unlikely to have exactly the same errors.

  3. Speaking now with my physical scientist hat on, I am confident no physicist would think that QED is THE CORRECT FINAL theory. We already know that it cannot be the last word. Quantum mechanics does not, at least yet, describe gravity correctly. We do not have a theory of quantum gravity. Any "final" theory of quantum electrodynamics is going to have to include the interaction of QED and gravity. This interaction may be very small, but it is certainly not zero. We physical scientists know that we don't have this yet. So Steele's assertion that QED "is simply true" cannot be correct.

    Richard Feynman, a hero of mine, was a self-promoter (and deservedly so), but I am sure that if he were still alive, he would agree that his baby, QED, for all of its marvelously accurate predictions, is not the final model. In his popular writings (like the book that Steele cites) he was often over-the-top, for very good pedagogical reasons.

  4. It is useful to say "all models are wrong, but some are useful". But it is also understandable that you don't like the statement if you are in the business of making probability statements that are conditional on the exact correctness of your model. Physicists generally do not make this mistake. In Feynman's "The Character of Physical law" it is stated, again and again, that models are never exact but obviously often useful. The same statement occurs in Bridgman, Wigner, Synge, and so on. I agree with Steele, however, that the statement can be easily misused to justify sloppy work.

  5. I love the saying and for someone in applied stats, the second statement is paramount. I prefer to say "all models are false" rather than "wrong".

  6. I think he misses the whole point of the saying, in that his example is a perfect one of the saying's validity.

    Of course maps are wrong — that is, they don't recreate all aspects of reality. Of course a good map is useful. The fact that he did not expect the map to perfectly re-create reality just shows that he knows what a map is, and knows that Box's saying is valid.

  7. I had always heard the phrase as "All models are false, some are useful."

    I always thought of it as a counter to the metaphysical sense of truth. For instance, I always hate it when commercials say "clinically proven" or when anyone says they have proven a scienfitic theory. Proof is for maths and alcohol!

    The map of a city is never TRUE because it leaves out the various cracks in the sidewalk, the corners of the true street aren't exactly 90 degrees, and the width of the streets aren't constant. But that's not to say this approximation of the city isn't useful.

  8. Just for clarification, at least one place where George Box talked about wrong vs. useful models was in a chapter he contributed to Robustness in Statistics, "Robustness in the Strategy of Scientific Model Building". The exact reference is the second section heading in that chapter, "ALL MODELS ARE WRONG BUT SOME ARE USEFUL" (original in caps). "All Models are Wrong … " follows directly after a section titled "THE NEED FOR SIMPLE SCIENTIFIC MODELS – PARSIMONY", and goes on to state:
    'Now it would be very remarkable if any system existing in the real world could be _exactly_ represented by any simple model. However, cunningly chosen parsimonious models often do provide remarkably useful approximations. For example, the law PV=RT … is not exactly real true for any real gas, but it frequently provides a useful approximation and furthermore its structure is informative since it springs from a physical view of the behavior of the molecules.
    For such a model there is no need to ask the question "Is the model true?". If "truth" is to be the "whole truth" the answer must be "No." The only question of interest is "Is the model illuminating and useful?" '
    In any case, when I discuss "All models are wrong but some are useful", I take the liberty of adding "but some are toxic," where "toxic models" justify counterproductive / dysfunctional / unethical behaviors. I think it's essential that people be able to distinguish between wrong/useful and wrong/toxic models – a model that's just wrong should still be used if it's sufficiently useful; a model that's wrong and toxic should be avoided at all costs.

  9. "All models are wrong" can be considered a model itself. A model of other models – a meta-model.
    Since it is a model, it is wrong (according to itself). Paradox. It is wrong, but useful :)

  10. I think the point of the saying is to remind people that a "model" is a synthetic approximation of reality and therefore subject to error. Many modelers actually fail to understand this. I once showed some ground truth (a barchanoid dune) that showed a predominate wind direction 50 degrees off from a GC model. The modeler told me the dune must be an artifact or anomaly. In other words, he chose his model as correct over reality.

    The saying is a warning to not stray into that mistake.

Comments are closed.