Freakonomics: What went wrong?

Kaiser and I tell the story. Regular readers will be familiar with much of this material.

We kept our article short because of space restrictions at American Scientist magazine. Now I want to do a follow-up with all the good stories that we had to cut.

P.S. Let me remind everyone once again that Freakonomics (the book and the blog) has some great stuff. Kaiser and I are only picking on Levitt & co. because we know they could do so much better.

P.P.S. Just to emphasize: our point that Freakonomics has mistakes is nothing new—see, for example, the articles and blogs by Felix Salmon, Ariel Rubenstein, John DiNardo, and Daniel Davies. The contribution of our new article is explore how it was that all these mistakes happened, to juxtapose the many strengths of the Freakonomics franchise (much of the work described in the first book but also a lot of what appears on their blog) with its failings. In some ways these contrasts are characteristic of social science research in general: a mix of careful assessment of assumptions (for example, in the use of instrumental variables analysis rather than simple comparisons to estimate causal effects) and casual storytelling.

And, as Kaiser and I discuss in our article, the “Freakonomics: What went wrong?” story has an additional factor: Levitt’s somewhat unstable mix of skepticism and trust, as indicated by, on one hand, his self-labeling as a “rogue economist” and, on the other, his experiences as a corporate consultant and student and teacher at elite universities. As we write:

We attribute many of these errors to the structure of the authors’ collaboration, which, from what we can tell, relies on an informal social network that has many potential failure points. In the original Freakonomics, much of whose content appeared originally in columns for the New York Times Magazine, the network seems to have been more straightforward: Levitt did the research, Dubner trusted Levitt, the Times trusted Dubner, and we the readers trusted the Times’s endorsement. In SuperFreakonomics and the authors’ blog, it becomes less clear: Levitt trusts brilliant stars such as [software executive Nathan] Myhrvold or [economics professor Emily] Oster, Dubner trusts Levitt, and we the readers trust the Freakonomics brand.

31 thoughts on “Freakonomics: What went wrong?

  1. “What’s more, having a major-league player as a father makes a boy “eight hundred times more likely to play in the majors than a random boy,” they write. If these factors are such crucial determinants of future stardom, what does this say about their theory that a star is made, not born?”

    I’m not sure this is a criticism of the “star is made, not born” story. Wouldn’t having an MLB father introduce the child to baseball much earlier, have him around elite athletes teaching him at all times, and put pressure on him to play baseball, rather than some other sport? “Having a father” isn’t implying someone is innately born a star the way I take it. I don’t see this as venturing very far from the initial implication…

    • Forgot to mention, I otherwise find this to be an instructive article. I, too, run into strange claims on the Freakonomics blog related to sports and sports economics (where my own research lies).

    • I agree with Millsy. I’m not a big fan of Freakonomics, but I appreciate Levitt and Dubner pointing out the fascinating seasonality of birthdays of professional athletes — boys born after before their sport’s age cutoff date are substantially more likely to have professional careers than boys born right before. I was skeptical at first, but my skepticism appears to have been wrong. This is an important thing to know. I appreciate Levitt and Dubner pointing it out to me.

        • I must admit I don’t see anything particularly “great” in the Freakonomoics book. It all looked like rather simple statistical analysis and basic sociological/psychological analysis, often dating from the 1960s and 1970s. I remember as I was there. :)

          I was disappointed in the book as it was so banal.

  2. Nice piece — esp the original. Freakonomics has had an interesting impact on academic economics. It did a great job selling economics to a broader audience — few of us can do this well — and I’m told (speculation!) we’ve seen a jump in undergrad econ majors as a result. But it seems to have generated a lot of work that’s more clever than thorough: hep b, abortion and crime, etc. These are wonderful examples of speculative thinking, which is one of the most interesting aspects of research, and they often make good stories. But as you and Kaiser say, they need to be labeled as speculation, not scientific conclusions. I’d still say we need more people who can communicate this well, you should look for your own Stephen Dubner.

  3. You left out the most obvious explanation: Levitt got away with his late 1990s abortion-cut-crime theory even though his work on that was so shoddy that he didn’t notice that the opposite of what he’d expected had actually happened: the homicide rate of males born in the half decade after abortion was legalized was vastly higher than the homicide rate of males born in the previous half decade — as I pointed out to him in our 1999 debate in Slate:

    http://www.slate.com/articles/news_and_politics/dialogues/features/1999/does_abortion_prevent_crime/_2.html

    If you can get away with theorizing about crime rates while not noticing the Crack Epidemic, what can’t you get away with? Even when Foote and Goetz of the Boston Fed showed in late 2005 that Levitt’s abortion-cut-crime finding was based on Levitt’s own programming error, that failed to dim his celebrity.

    As it turned out, what you can’t get away with is what Levitt did in SuperFreakonomics: display skepticism about certain aspects of Global Warming theory. Levitt was the media golden boy up until then, but Global Warming is too sacred.

    • Steve,

      What peer-review cites do you have for this? It sounds d*mning to his case, but when I googled this, the only items on the net were your writings.

      Barry

  4. It’s unfair to denigrate Levitt and Dubner without comparing the reliability of the Freakonomics brand to that of their chief rival, the Malcolm Gladwell of The New Yorker Brand. My view is that Levitt and Dubner are more trustworthy than Gladwell by a comfortable margin.

    • The purpose of our article was not to denigrate the Freakonomics authors but to explore how they went wrong when they did. I’d be fascinated to see a similar article about Gladwell. In this case the enablers might be Gladwell’s editors at the New Yorker. (As we’ve discussed on occasion on this blog, even the New Yorker’s fabled fact-checkers allow a few mistakes to creep in to the magazine on occasion.)

    • Like Andrew said, the point of our article is to praise the impact they’ve had on popularizing statistics while pointing out difficulties and challenges of doing so. A similar piece on Gladwell’s oeuvre would be nice asl well. In fact, I have previously posted on Blink (see here and here.).
      For the reason you stated, we have more of an interest in the Freakonomics franchise. We should recognize that Gladwell is not a scientist (nor does he proclaim to be one) while one half of the Freakonomics team is a scientist, and a well-respected one.

      • Gladwell is the #1 conduit to the public for academics’ bad ideas.

        He doesn’t do simple reality checks on theories that smart-seeming people tell him because he’s just not cynical enough. He really, truly admires all these people he writes about and believes they are all brilliant, even though their theories often contradict each other. (That’s why his bestseller Blink made no overall sense whatsoever.)

        As Gladwell wrote in 2006 after breathlessly retailing a couple of economists’ dubious explanation of Ireland’s recent prosperity and getting shot down by commenters on his own blog:

        “I will confess to having a slightly reverential attitude toward academia. I’m the son of an academic. Much of my writing involves taking academic research and trying to translate it for a more general audience. And I’ve always believed that if you set out to write about the work of academic specialists, you have a responsibility to treat that work with respect– to acknowledge your own ignorance and, where appropriate, defer to the greater expertise of others.”

        • (Steve S. quotes Gladwell saying:)
          > “I’ve always believed that if you set out to write about the work of academic specialists, you have a responsibility to treat that work with respect– to acknowledge your own ignorance and, where appropriate, defer to the greater expertise of others.”

          This sounds like my approach to climate science. The differences between climate science & Gladwell subject matter being, 1) garden variety climate communication is loaded with charlatans to a degree that statistics communication isn’t, and 2) compared to those charlatans, non-contrarian climate scientists are utter founts of wisdom. So, uncritically quoting non-contrarian climate scientists does in fact raise the bar, albeit not as high as it’d be raised if we took the time to check everything they say against their peers.

          (or so I think)

  5. Excellent polemic. What you miss, if anything is the Churnalism driver. Leveraging their initial best-seller, the Steves created a franchise, and to murder a simile, like Mickey D’s, franchises require tons of raw meat, quality is not necessary Any cow to be slaughtered is welcome and they have to grind up a lot of it. Careful testing and research are optional in churnalism. Also maintaining an illusion of omnipotence (see Gingrich, Newt and Emperor, New Clothes)

  6. Hey Andrew,
    Have you seen the most recent journal article written by Levitt?

    It is called:
    “THE ROLE OF SKILL VERSUS LUCK IN POKER: EVIDENCE FROM THE WORLD SERIES OF POKER”

    From my perspective it was an absolute train-wreck and I’m not exactly sure (a) what he is studying and (b) why it would be important. If other comm enters have read it, I would love to hear what you think as well.

    Here is a link to the article:
    http://www.nber.org/papers/w17023.pdf?new_window=1

  7. ‘But you don’t need to believe in astrology to realize that the two cited probabilities are not the same. A .300 batting average is 50 percent better than a .200 average. In such a competitive field, the difference in batting averages between a kid who makes the majors and one who narrowly misses out is likely to be a matter of hundredths or even thousandths of a percent. Such errors could easily be avoided.’

    I’d have to give credit to Levitt and Dubner here. I have no clue what the cumulative batting averages of kids born in July and August are, or how to even calculate it. It is reasonable to making the majors as a measurable endpoint of baseball skill, and also reasonable journalistic flourish to refer to that as ‘50% better at hitting a big-league curveball.’ Indeed, batting average would be just as poor a measure. After all, Babe Ruth’s batting average of .342 is only 59% better than Mario Mendoza’s batting average of .215.

    • Can someone show me the math on this?

      I have no idea of how batting averages are calculated and I don’t see this at all.

    • You don’t want to use batting average here. On base percentage is a much better alternative. Batting average is pretty much going by the way side. If you use OBP the difference between Mendoza and Ruth becomes more pronounced. At least with OBP you are including walks, and Ruth gets walked a ton more than Mendoza. Considering Ruth also has the second highest OBP of all time only behind Ted Williams. If your going by batting average (an almost defunct statistical measure), then your argument stands. At the end of the day we care more about how often you get on base (including walks) than whether you hit the ball well (batting average). I’m not saying the same argument can’t be made using OBP, I’m saying it should.

      • I’d argue that a .300 hitter is much, more more than 50% better. In terms of likely market value, and since baseball owners aren’t idiots [assumption!] this is true of their value to the team as well. After all, you can’t have 1.5 .200 hitters bat instead of a .300 hitter.

        According to the local paper, Albert Pujols’ new 10 year contract is 7 times the total payroll of the Kansas City Royals. http://articles.chicagotribune.com/2011-12-09/news/ct-talk-pujols-254-million-1209-20111209_1_contract-ipad-wi-fi-and-3g

        Perhaps relevant to this argument about birth months is the persistent rumor that Pujols is actually older than he admits, and therefore has fewer good years left. Dominican prospects want to claim to be younger than they are, because then they are evaluated versus a different cohort. Performance at the 90th percentile if you are age 18 might look like the 99th percentile if you can be evaluated against 16 year olds, for example.

  8. Batting Average = Hits/(At bats – walks- sacrafices- hit by pitcher)

    The last three categories represent opportunities to hit that did not result in a hit but don’t count for the batting average because they are positive outcomes for the team (the last can be painful tho)

    • This is wrong.

      Batting Average= Hits/At Bats

      You don’t take into account walks or sacrifices or hits by pitcher in the denominator. Otherwise everyone’s batting averages would be huge.

      Hence the problem with batting average it does not take into account walks. Not to shut you down, but fix your post.

  9. Pingback: Freakonomics: Why ask “What went wrong?” « Statistical Modeling, Causal Inference, and Social Science

  10. Pingback: Brett Keller » Blog Archive » Monday Miscellany

  11. Pingback: A kaleidoscope of responses to Dubner’s criticisms of our criticisms of Freaknomics « Statistical Modeling, Causal Inference, and Social Science

Comments are closed.