Skip to content

Derman, Rodrik and the nature of statistical models

Interesting thoughts from Kaiser Fung.

Derman seems to have a point in his criticisms of economic models—and things are just as bad in other social sciences. (I’ve criticized economists and political scientists for taking a crude, 80-year-old model of psychology as “foundational,” but even more sophisticated models in psychology and sociology have a lot of holes, if you go outside of certain clearly bounded areas such as psychometrics.)

What can be done, then? One approach, which appeals to me as a statistician, is to more carefully define one’s range of inquiry. Even if we don’t have a great model of political bargaining, we can still use ideal-point models to capture a lot of the variation in legislative voting. And, in my blog post linked to above, I recommended that economists forget about coming up with the grand unified theory of human behavior (pretty impossible, given that they still don’t want to let go of much of their folk-psychology models) and put more effort into their central competence of analyzing prices, capital flows, and so forth.

At the same time, governments, businesses, and other organizations need to make decisions involving macroeconomics. So we can’t just give up. And, in my own applied statistical work, I use imperfect models all the time.

Here’s what Kaiser writes:

The insurmountable challenge of social science models, which constrains their effectiveness, is that the real drivers of human behavior are not measurable. What causes people to purchase goods, or vote for a particular candidate, or become obese, or trade stocks is some combination of desire, impulse, guilt, greed, gullibility, inattention, curiosity, etc. We can’t measure any of those quantities accurately.

What modelers can measure are things like age, income, education, past purchases, objects owned, etc. Nowadays, we can log every keystroke you type on your smartphone. That models are even half-accurate is due to the correlation of these measured quantities with the hidden drivers of our behavior but this correlation is only partial.

Now add to that, the vagaries of human behavior.


  1. Clark Andersen says:

    This may be naive, but it sounds to me like there may be a role for factor analysis in addressing this problem. It seems analogous to that of estimating the “G” factor associate with intelligence. The factors of interest are not directly measurable, but may be inferred from other things which can be measured.

  2. Jonathan says:

    Nice insight as always Andrew.

  3. MAYO says:

    I am in need of getting clearer on the controversies in modeling and methodology in developmental economics—is that something anyone here is in the know about? It seems the advocates of RCTs tend to overpromise, and to cause researchers to feel they are protected from many model misspecifications that still exist. Critics, on the other hand, are sometimes thought to be denouncing RCTs altogether. The whole issue of dealing with active agents introduces major contrasts with medical research.

  4. K? O'Rourke says:

    Maybe it’s like growing old – there just aren’t appealing alternatives.

    Two contrasts of strategies for dealing with the insurmountable opportunities of models in difficult fields.

    Ferdinand de Saussure: As for economics, the models are about how people value things and as you discuss theses models, people will change how they value things. So he chose to work in linguistics, where models are about how terms affect how other terms are interpreted by people. [Changed to a field where the model was way less wrong?]

    Sigmund Freud: In psychology [in particular, his Project for a Scientific Psychology], scientific models for psychology are going to remain way too wrong during my lifetime – but I have no intention of changing disciplines. [The one eyed man will be King in the land of the blind?]

  5. Dave Backus @ NYU says:

    Modern macroeconomics, as developed by Lucas, Sargent, and others, is concerned precisely with the instability of macroeconomic relations — correlations between variables, for example. They show, in lots of contexts, that if you change (say) monetary policy, a lot of other things will change with it. You could say the same thing about asset returns. If a subset of investors decides to invest differently (the quants, for example), it’s not hard to imagine that the statistical properties of asset returns will change as a result. Asset prices don’t follow immutable physical laws, they’re the result of investors’ actions.

    None of this is new, the tools date back to the 1970s and before. The trick is to disentangle the various pieces so that we can do something beyond throwing up our hands. That’s a lot harder, and there’s a range of opinion how much progress has been made.

    So if I read Derman right: yes, economics and finance are different from physics. And Rodrick: yes, it’s true, there’s a lot of ambiguity about how the economy works. As my friend Kydland likes to say: Be humble about what you think you know.

  6. Will says:

    I find the original post a bit confusing.

    On the one hand, it has an uncomfortable amount of data fundamentalism, e.g. the easy movement between “not measurable” and “can’t be measured accurately”. Your points about psychometrics are well taken: moves like the one above are like measurement theory never happened. Indeed, it’d apparently be useless since it’s based on the idea that direct observation of interesting quantities is seldom possible. So, so much for ideal point models…

    On the other hand, he also claims that “what modelers can measure are things like [...] education [...]“. I guess he means that we can measure things like years spent in educational establishments, and number of degrees attained. But these would seem to have about the same indirect relationship to education, i.e. how (well) educated a person is, as whatever proxies we might find for greed, impulsivity, etc.

    • Kaiser says:

      Will: Let me clarify. My post does not say models based on demographics are not useful. I’m just saying that there will always be a (huge) gap between what is measured and what influences behavior, and modelers should acknowlege that. Say in a factor analysis, one might interpret a factor to represent “curiosity” but that’s an interpretation, not measurement.

      As for your second point, by “education”, I mean the typical Census-based variables that are commonly used by lots of modelers. And of course, “education” does not measure “learning” or “educated-ness”.

      • Will says:

        Thanks for the clarification. I guess you’d say then that Andrew’s example of ideal point estimation (essentially factor analysis with binary variables) is “not measurement” either and in particular that does not measure ideological position? If it doesn’t then bargaining models that rely on such ideal point information may just be hopelessly interpretative. I’ve certainly heard that argued…

  7. Kaiser says:

    Clark: your instinct is right. There are a lot of factor analysis, structural models, latent class models, etc. used in social science. However, we face the same problems as those in the field of intelligence. What is G anyway? Does it actually exist? Like Andrew and Dave and others have said here, we have to be careful about what are assumptions, which parts are unverifiable, etc. Above all, each model is one abstraction of reality, and there can’t be one dominant model because all models have flaws.