Palko’s theory is that Levitt and Dubner’s most logical decision, from a cost-benefit perspective, was to avoid peer review (here I’m using the term generally, considering statisticians such as Kaiser Fung as “peers” whether or not the reviewing is done in the context of a formal journal submission) so as to get a marketable product out the door with minimal effort:
I [Palko] am not saying that Levitt and Dubner knew there were mistakes here. Quite the opposite. I’m saying they had a highly saleable manuscript ready to go which contained no errors that they knew of, and that any additional checking of the facts, the analyses or logic in the manuscript could only serve to make the book less saleable, to delay its publication or to put the authors in the ugly position of publishing something they knew to be wrong.
I think this theory has a lot going for it, although maybe it could be framed in a slightly more positive way. Consider my favorite of Kaiser’s comments on Freakonomics 2:
Would like to see a study of Internet substituting pimps. As it stands, this is an assertion without proof.
Palko’s reasoning definitely applies here. Levitt and Dubner already had a chapterful of material on prostitution, and they’re not preparing a treatise on the subject, so no point in going further. Similarly, the notorious offhand remark on drunk driving (which they attribute to “brilliant economist Kevin Murphy”) is worth a page. No point in looking at it in detail and ruining the story.
Overall, though, I’d still go with a slightly different theory, which is that Levitt and Dubner believe that everything in their book is correct. Well, not quite. At the end of chapter 1, they have a disclaimer that they are writing the book to start conversations, not to end them, and that they want to stir up controversy and discussion. Still, I think they’re a bit more gullible credulous than professional statisticians such as Kaiser or myself. The problem, I think, is that they trust their friend/experts too much. Trusting friends and experts makes a lot of sense, I think, but if you’re not careful it can lead to some silly mistakes.
I have to be careful when making this claim, given the controversy that ensued after I speculated on how Dubner could’ve endorsed the ridiculous (to me) claim that “most (if not all!) deaths are to some extent ‘suicides.’”
More principles of economics
The other thing that’s going on, I suspect, is division of labor. To start with, I doubt Dubner would be checking the sorts of statistical points that Kaiser is raising–really, that would be the job of Levitt, who’s one of the most qualified people in the world to evaluate quantitative claims in social science. (For example, I think that if Levitt had ever carefully read the papers by Monica Das Gupta, he wouldn’t have been so quick to jump to conclusions about the causes of China’s missing girls.) But, to continue with the division-of-labor argument, I don’t think Levitt thinks it’s his job to check these sorts of statistical arguments either. He’ll trust his friends and colleagues–after all, they know the various topics better than he does, right?
Usually it makes sense to trust the experts–especially those that you personally know and trust–but not so much if you’re writing a whole book full of counterintuitive examples. It’s selection bias: people’s most controversial statements are the most likely to be mistaken. If these statements never get checked, then you have little more than a set of press releases linked with engaging commentary. Perhaps Kaiser Fung could be added as a coauthor for Freakonomics 3, to add some of that close reading that could make all the difference?
P.P.S. I better repeat my disclaimer from last year: I’ve been picking on Freakonomics a lot recently, but really this is the result of selection bias: when Freakonomics has material of its usual high quality, I don’t have much to add, and when there’s material of more questionable value, I notice and sometimes comment on it. Those of us who’ve contributed to the burgeoning “what went wrong with Freakonomics 2?” literature are doing so only because we believe its authors could do better, if they were to put in the effort.