We kept our article short because of space restrictions at American Scientist magazine. Now I want to do a follow-up with all the good stories that we had to cut.
P.S. Let me remind everyone once again that Freakonomics (the book and the blog) has some great stuff. Kaiser and I are only picking on Levitt & co. because we know they could do so much better.
P.P.S. Just to emphasize: our point that Freakonomics has mistakes is nothing new—see, for example, the articles and blogs by Felix Salmon, Ariel Rubenstein, John DiNardo, and Daniel Davies. The contribution of our new article is explore how it was that all these mistakes happened, to juxtapose the many strengths of the Freakonomics franchise (much of the work described in the first book but also a lot of what appears on their blog) with its failings. In some ways these contrasts are characteristic of social science research in general: a mix of careful assessment of assumptions (for example, in the use of instrumental variables analysis rather than simple comparisons to estimate causal effects) and casual storytelling.
And, as Kaiser and I discuss in our article, the “Freakonomics: What went wrong?” story has an additional factor: Levitt’s somewhat unstable mix of skepticism and trust, as indicated by, on one hand, his self-labeling as a “rogue economist” and, on the other, his experiences as a corporate consultant and student and teacher at elite universities. As we write:
We attribute many of these errors to the structure of the authors’ collaboration, which, from what we can tell, relies on an informal social network that has many potential failure points. In the original Freakonomics, much of whose content appeared originally in columns for the New York Times Magazine, the network seems to have been more straightforward: Levitt did the research, Dubner trusted Levitt, the Times trusted Dubner, and we the readers trusted the Times’s endorsement. In SuperFreakonomics and the authors’ blog, it becomes less clear: Levitt trusts brilliant stars such as [software executive Nathan] Myhrvold or [economics professor Emily] Oster, Dubner trusts Levitt, and we the readers trust the Freakonomics brand.