Freakonomics: Why ask “What went wrong?”

A friend/colleague sent me some comments on my recent article with Kaiser Fung on Freakonomics. My friend gave several reasons why he thought we were unfair to Levitt.

I’ll give my reply (my friend preferred that I not quote his email, but you can get a general sense of the questions from my answers). But first let me point you to my original post, Freakonomics 2: What went wrong?, from a couple years ago, in which I raised many of the points that ultimately went into our article.

And here’s my recent note. (I numbered my points, but the email I was replying to was not numbered. This is not a point-by-point rebuttal to anything but rather just a series of remarks.)

1. Both Kaiser and I are big fans of Freakonomics. It’s only because Levitt can (and has) done better, that we’re sad when he doesn’t live up to his own high standards. If we didn’t convey this sense of respect in our American Scientist article, that is our failing.

2. I think it was at best tacky and at worst irresponsible for Levitt to repeatedly promote the work of his economist colleague on sex ratios and then, when the work was retracted, praise the economist who had made the mistake, without ever acknowledging the scientists who had been right all the time.

3. I think it was at best tacky etc. for Levitt to non-respond to Pierrehumbert’s comments by labeling these as an “intentional misreading” and then, referring to his own writings on the subject, to write, “I’m not sure why that is blasphemy.” Nobody ever said Levitt’s writings were “blasphemy” ant it just seems silly to me for Levitt to say that.

4. I have come across some other of Myhrvold’s writings and he seems like a bit of a crank. He has a habit of applying the concept of albedo in all sorts of inappropriate settings.

5. Considering points 2-4 above, I think it’s fair to say that in his recent writings, Levitt has been a bit too quick to trust the work of his friends.

6. There are also a bunch of other examples that we didn’t have room for in our article (we had to cut a lot for space limitations), for example in October 2008 endorsing his U of C colleague Casey Mulligan’s “economic arguments as to why things are not so bad.” (Mulligan wrote, “The current unemployment rate of 6.1 percent is not alarming.”) I can’t try to evaluate Mulligan’s arguments, but my point here is that Levitt is perhaps too quick to trust his friends. I have never met Levitt, but I have heard from others that he’s a very nice guy. And in his writings, he portrays himself in a humble manner. The flip side of this, however, is that he’s a bit too uncritical of friends. Perhaps if Levitt had a higher self-regard, he’d be more willing to apply critical standards to others.

7. Levitt is the one who called himself a rogue (or at least acquiesced in the portrayal). I don’t think there’s anything wrong with us pointing out the contradictions and difficulties that arise from this role. For that matter, I don’t think there’s anything wrong with not being a rogue. I have the same sort of Harvard-MIT background that Levitt has. I freely admit to not being a rogue, even though there have been times and places (e.g., UC Berkeley in the early 1990s) where I happened to be an outsider. Overall, I’ve always been part of the establishment, and there are a lot of advantages to that, even beyond being paid well. Levitt’s part of the establishment too. He’s used his advantages well, and I don’t begrudge him that one bit.

8. To return to point 1 above, I think Levitt did a great service by writing Freakonomics (and doing the research on which the book was based) and also continuing with the Freakonomics blog. I particularly like how Levitt and Dubner convey the intellectual excitement of research and the back-and-forth between theory and data. Commendably, they don’t put themselves above the fray but rather are open about the active nature of scientific theorizing.

But then, when they falter, they disappoint me. For example, getting back to item 2 above, Levitt and Dubner were sitting on a wonderful story about how a smart economist made a mistake and how that economist refused to admit error for a couple of years even when the experts in the field showed evidence that she was wrong. But instead Levitt flattened the story by making the economist the hero and never mentioning the others. This is Washington-chopping-down-the-cherry-tree history, it’s a missed opportunity, and I think it’s disrespectful to the researchers in that field who were right all along.

To put it another way:

About a year ago, I gave my talk, “Of Beauty, Sex, and Power,” at the meeting of the National Association of Science Writers. At one point I mentioned Freakonomics and the audience groaned. Steve Levitt is not a popular guy with this crowd. And that’s the typical reaction I get: “Freakonomics” is a byword for sloppy science reporting, it’s a word you throw out there if you want an easy laugh. Even some defenders of Freakonomics nowadays will say I shouldn’t be so hard on it, it’s just entertainment.

Now go back a few years. In 2005, Freakonomics was taken seriously. It was a sensation. Entertaining, sure, but not just entertainment—rather, the book represented an exciting new way of looking at the world. There was talk of the government hiring Levitt to apply his Freakonomics tools to catch terrorists.

That’s what Kaiser and I meant when we asked “What went wrong?” Freakonomics was once a forum for a playful discussion of serious, important ideas; now it’s more of a grab-bag of unfounded arguments. There’s some good stuff there but seemingly no filter.

It’s not news that Freakonomics has problems. See, for example, the reviews by Felix Salmon, Ariel Rubenstein, John DiNardo, and Daniel Davies.

In reporting and discussing some cases of questionable judgment on the part of the Freakonomics team, Kaiser and I were not trying to shoot down Freakonomics but to explore what went wrong: What moved them on the track from serious popular science to a walking punch line?

We identify the problem as one of quality control: lots of material and nobody checking it. That’s how you get an uncritical trust in an economist with a wrong theory of sex ratios, an albedo-obsessed software executive, an economist who claimed in October 2008 that “things are just not that bad,” Daryl Bem, and . . . a certain Reader in Management at the London School of Economics.

My guess is that Levitt takes his academic writing seriously but think of the Freakonomics books and blogs as just for fun. Although Levitt and Dubner could probably hire someone to fact-check their books and blogs, they don’t really see the point, since their goal is to get people thinking rather than to get the right answer.

But if they would admit their mistakes (rather than either ignoring them or going on the offensive when people point out problems), maybe they could move back to their former role of serious communicators of social science. This is what Kaiser and I talk about in our article: the tensions between accuracy and readability.

17 thoughts on “Freakonomics: Why ask “What went wrong?”

  1. What kills me is, finding out that you were wrong is just part of discovering new things. And learning how and why you were wrong is even more fun and more useful. There ought not be a tradeoff, for writers as lively and sharp as Levitt and Dubner, between accuracy and readability.

    It’s a shame to see them risk degenerating into a cautionary tale.

  2. I’m reminded of my reaction to Charles Murray’s book of a few decades ago, “Losing Ground.” He looked at crime policy, education policy, welfare policy, poverty policy, etc., in that book. I thought he was right on in the other fields, but in criminal justice (my field) I felt he misused the data and got it wrong. I then found out that people in the poverty field at the University of Wisconsin felt that he had done a useful job on all areas he looked at except poverty, where he misused the data and got it wrong. Sometimes, as in The Music Man, you’ve got to know the territory — and playing with data without knowledge of its origins and faults leads you to make sometimes egregious mistakes.

    • That reminds me of a story someone told about Immanuel Velikovsky’s “Worlds in Collision”. They thought the astrophysics, history, etc. was really interesting, but the chemistry was crap – hydrocarbons falling out of the sky, turning to carbohydrates, and becoming the manna that fed the Israelites in the desert! Then they talked to a colleague, who was an astrophysicist, who said the astrophysics was terrible, but he thought the history and the manna stuff was really interesting. So they went off to talk to a historian, who said…

      • I find these reactions interesting because my default reaction is exactly the opposite. If someone is wrong about something I know about, I assume they are probably wrong about their other assertions. For example, stock analysts’ pronouncements about the industry I work in are always wrong, so I never trust stock analysts, no matter what industry they talk about. Or more to the point, the stupid and evil assertion that drunk walking is more dangerous than drunk driving leads me to question everything the Freakonomics crowd does. Btw, I call it stupid because 30 minutes of research on the internet would lead any intelligent person to realize that they didn’t know what they were talking about. It’s evil because its only possible result is to get someone killed prematurely. I honestly don’t know how the Freakonomics people live with themselves.

    • MikeM: But I think you have a bigger point – as CS Peirce once wrote something like “Good thing that you die otherwise you would eventually learn that you were wrong about everything [in all areas].”

      The high road seems to be in accelerating the process of getting less wrong in all areas.

      And not being defensive, even for friends.

    • Actually, K, in a sense we’re dealing with Simpson’s paradox. I look at the stuff I did in the past, mathematical modeling of criminal justice data (that I took as the truth), and now realize that no matter how finely I sliced it at the time, it was still baloney. Now I spend my energy for the most part on cleaning data, so that others who follow me will do a better job.

  3. Of course, what I mean is a corollary to Simpson’s paradox — the more deeply you dig into the data, the closer you get to “truth.”

  4. Tough problem. There’s great value in writing for a broader audience than we academics usually do. But the people who do it are invariably sloppy. That’s true of a lot of Levitt’s academic work, too. Also (apparently) of Steven Pinker. Not clear anyone can do both, no one has enough time, so we’re stuck getting one or the other. The whole thing leaves me wildly ambivalent, but I do think Freakonomics has generated a lot of interest, and that’s a good thing.

  5. Pingback: Blogging, polemical and otherwise « Statistical Modeling, Causal Inference, and Social Science

  6. Pingback: A kaleidoscope of responses to Dubner’s criticisms of our criticisms of Freaknomics « Statistical Modeling, Causal Inference, and Social Science

Comments are closed.