In response, I thought it would be interesting to go “meta” here by considering all the different ways ways that I could reply to Dubner’s criticism of our criticism of his writings. (In the same post, Dubner also slams Chris Blattman (or, as he calls him, “a man named Chris Blattman”), but I won’t get into that here.)
In reacting to Dubner, I will give several perspectives, all of which I believe. Usually in writing a response, one would have to choose, but here I find it interesting to present all these different perspectives in one place.
I understand where Dubner is coming from. We say some positive things and some negative things about his work, but it’s natural for him to focus on the negative. I’m the same way myself, probably almost all authors are. In not replying to every point in Dubner’s post, I am not implying that I agree with the items not mentioned; rather, I’m relying on what Kaiser and I have already written. We said our piece, Dubner’s had his reply, and the reader can work it out from there.
2. Show respect.
As I wrote to Dubner last week:
Both Kaiser and I greatly admire Freakonomics, especially in the way that in your books and your blog you engage the reader in the struggles and excitement of discovery in the social sciences. Also, we share your interest in topics ranging from sports to baby names to criminology. Where we do criticize your writings, it is always from the perspective that your work has set a high standard. As sometime popularizers ourselves, Kaiser and I are well aware of the challenges of communicating topics of scientific debate to general audiences.
In his recent post, Dubner writes that we “come to bury, not to praise” his work. I regret that we gave that impression. As noted, we are fans of Freakonomics. I can’t speak for Kaiser but I continue to read the Freakonomics blog with interest. My disappointment at some of the low-quality items does not remove my appreciation of the good stuff, and I feel bad that we did not emphasize this more in our article.
Here it is again (from a couple years ago).
3. Step back.
Dubner lives in different worlds than those of Kaiser and me. (Levitt is in between, with one foot in the publishing/media world and the other in academia.) To the millions of readers of his books and blogs, Levitt and Dubner are the kings (and rightly so, the’ve done some great stuff), and Kaiser and I have the status of moderately-annoying gnats.
But I suspect Dubner realizes that, outside of his circle, he and Levitt have some credibility problems. They have fans but a lot of non-fans too. As I wrote a couple months ago:
About a year ago, I gave my talk, “Of Beauty, Sex, and Power,” at the meeting of the National Association of Science Writers. At one point I mentioned Freakonomics and the audience groaned. Steve Levitt is not a popular guy with this crowd. And that’s the typical reaction I get: “Freakonomics” is a byword for sloppy science reporting, it’s a word you throw out there if you want an easy laugh. Even some defenders of Freakonomics nowadays will say I shouldn’t be so hard on it, it’s just entertainment.
Now go back a few years. In 2005, Freakonomics was taken seriously. It was a sensation. Entertaining, sure, but not just entertainment—rather, the book represented an exciting new way of looking at the world. There was talk of the government hiring Levitt to apply his Freakonomics tools to catch terrorists.
That’s what Kaiser and I meant when we asked “What went wrong?” Freakonomics was once a forum for a playful discussion of serious, important ideas; now it’s more of a grab-bag of unfounded arguments. There’s some good stuff there but seemingly no filter.
This is what I’m talking about. When a roomful of science reporters treats you like a punch line, the problem isn’t with statisticians Gelman and Fung, or with economists Ariel Rubinstein and John DiNardo, or with bloggers Felix Salmon and Daniel Davies (to name several people who have published serious criticisms of Freakonomics). There are deeper problems, some clue of which might be found by reading all these critiques with an eye to learning rather than mere rebuttal. Don’t get distracted by your fans on the blog—consider that room full of science writers! Try to recover the respect of Felix Salmon and Daniel Davies; that would be a worthy goal.
As noted above, Dubner seems to misunderstand the purpose of our article. We think Freakonomics has some great stuff but it also seems (to us) to lend an air of authority to some speculations and some outright mistakes. We had to keep our article short to fit the constraints of American Scientist magazine, so we thought it best to detail a few of the problems and then discuss how it all could’ve happened.
Dubner asks why we did not include other examples from the Freakonomics book or from their radio broadcasts. The answer is that we did not have space to include all the examples we had already found. We were not looking for more. Our point was not that X% of Freakonomics articles had problems, our point was that some avoidable errors had gotten through.
In another confusion of motivation, Dubner describes it as “weaselly” when Kaiser and I wrote, “Although there’s no way we can be sure, perhaps, in some of the cases described above, there was a breakdown in the division of labor when it came to investigating technical points.” It looks weaselly to Dubner, but to a statistician such as myself, it’s caution. Dubner compares our writing unfavorably to that of freshman composition students. Fair enough: he’s the professional writer, we are not. I fear it would upset Dubner even further to learn that Kaiser and I are generally considered to write better than the average statistician! I’ll just say that we do our best and that we would welcome the constructive suggestions of any professional writer who cares to give us advice. Until then, I recommend that Dubner accept that academics tend to write in a certain stilted way, and there’s no need to attribute weaseliness to our awkward prose stylings.
To get back to the specific point, we suspect there were breakdowns in the division of labor, but we can’t be sure, so that’s what we wrote. I’ve never met Steven Levitt but I’m pretty sure that he has the technical ability to, say, evaluate a statistically-flawed claim about beauty and sex ratios. But he’s a busy man and doesn’t necessarily have the time for it. That’s an example of a breakdown in the division of labor.
Here are a few of the other things that have been presented uncritically on the Freakonomics blog over the past several years:
– Casey Mulligan’s claim in October 2008 that the economy is not that bad: “The current unemployment rate of 6.1 percent is not alarming.”
– The ESP research of Daryl Bem. In no uncertain terms, “Daryl Bem has demonstrated ‘numerous “retroactive” psi effects – that is, phenomena that are inexplicable according to current scientific knowledge . . .'”
– A report, unqualified by any skepticism, of a dubious claim that “The companies Tiger Woods endorsed — and their shareholders — are feeling the negative effects of his extramarital affairs.”
Any of these (along with the other examples that have come up on this blog and elsewhere) is understandable, but when you put it all together, the message is that you can’t trust what you see in Freakonomics. It’s often interesting, often provocative, sometimes avoidably wrong. There’s an on switch but no off switch, no skeptical Felix Salmon voice in the background saying, “Hey, wait a minute? Are you sure about that?”
Levitt and Dubner can do better—I know they can do better, I’ve seen it in their best work—and I hope that they will be motivated by our critiques, along with those of Rubinstein, DiNardo, Salmon, and Davies, to be more careful about trusting the latest claims of albedo-focused billionaires, ESP researchers, politically incorrect sociologists, and various other rogues who come to them with compelling stories. Dubner is a professional journalist with a proven talent for inspiring millions of readers with the excitement of discovery; I am a sour statistician, trained to be skeptical. I’m sure each of us could stand to be a bit more like the other.