Reed Abelson and Gardiner Harris report in the New York Times that some serious statistical questions have been raised about the Dartmouth Atlas of Health Care, an influential project that reports huge differences in health care costs and practices in different places in the United States, suggesting large potential cost savings if more efficient practices are used. (A claim that is certainly plausible to me, given this notorious graph; see here for background.)
Here’s an example of a claim from the Dartmouth Atlas (just picking something that happens to be featured on their webpage right now):
Medicare beneficiaries who move to some regions receive many more diagnostic tests and new diagnoses than those who move to other regions. This study, published in the New England Journal of Medicine, raises important questions about whether being given more diagnoses is beneficial to patients and may help to explain recent controversies about regional differences in spending.
Abelson and Harris raise several points that suggest the Dartmouth claims may be overstated because of insufficient statistical adjustment. Abelson and Harris’s article is interesting, thoughtful, and detailed, but along the way it reveals a serious limitation of the usual practices of journalism, when applied to evaluating scientific claims.
The problem is that Abelson and Harris apply a shotgun approach, shooting all sorts of criticisms at the study without a sense of what makes sense and what doesn’t. For example, they write:
But while the research compiled in the Dartmouth Atlas of Health Care has been widely interpreted as showing the country’s best and worst care, the Dartmouth researchers themselves acknowledged in interviews that in fact it mainly shows the varying costs of care in the government’s Medicare program. Measures of the quality of care are not part of the formula.
For all anyone knows, patients could be dying in far greater numbers in hospitals in the beige [low-spending] regions than hospitals in the brown [high-spending] ones, and Dartmouth’s maps would not pick up that difference. As any shopper knows, cheaper does not always mean better.
Setting the maps aside, could it really be true that “patients could be dying in far greater numbers in hospitals in the beige regions than hospitals in the brown ones”?? I really doubt that, and I’m pretty sure that this has been checked. I mean, that’s an obvious thing to look at. And, in fact, later on in the news article, the authors write that, “a 2003 study found that patients who lived in places most expensive for the Medicare program received no better care than those who lived in cheaper areas.”
So what’s the deal with “For all anyone knows, patients could be dying in far greater numbers in hospitals in the beige regions than hospitals in the brown ones”? Are Abelson and Harris saying there was a problem with the 2003 study, or that there have been big changes in 2003, or . . . ? It’s certainly possible that I’m missing something here myself!
Abelson and Harris then write:
Even Dartmouth’s claims about which hospitals and regions are cheapest may be suspect. The principal argument behind Dartmouth’s research is that doctors in the Upper Midwest offer consistently better and cheaper care than their counterparts in the South and in big cities, and if Southern and urban doctors would be less greedy and act more like ones in Minnesota, the country would be both healthier and wealthier.
But the real difference in costs between, say, Houston and Bismarck, N.D., may result less from how doctors work than from how patients live. Houstonians may simply be sicker and poorer than their Bismarck counterparts. Also, nurses in Houston tend to be paid more than those in North Dakota because the cost of living is higher in Houston.
Huh? One of the striking things about the cost-of-care map is how little it corresponds with cost of living. The high-cost regions include most of Texas, just about all of Louisiana, Mississippi, Arkansas, Oklahoma, and Tennessee (as well as some more expensive places such as the Bosnywash area and much of California.) There may be a lot of problems with this study, but I can’t imagine that one of these problems is a lack of accounting for the (relatively) high cost of living in Houston.
How can this sort of energetic reporting be improved?
I picked on a couple of funny things in the Abelson and Harris, but overall I think they’re doing a service by examining an influential report. My purpose is not to slam them but to suggest how they could do better.
Their fundamental difficulty, I think, is the challenge of writing about a technical topic (in this case, statistics) without the relevant technical experience. (I was going to say “technical training,” but it’s not clear that Stat 101 or even a Ph.D. in statistics or economics will really teach you how to get a sense of perspective when evaluating quantitative claims.) The usual journalistic solution to reporting a technical controversy is to retreat to a he-said, she-said template, quoting experts on both sides and letting the reader decide. To their credit, Abelson and Harris try to do better than this by raising statistical objections on their own–but this strategy can backfire, as in the two examples above.
What should they (and other similarly-situated reporters) do next time? To start with, I’d recommend getting more input from qualified outsiders. Abelson and Harris do this a bit, with a quote from health economist David Cutler–but they only give Cutler one offhand sentence. I’d be curious to hear what Cutler would say about the claim that “for all anyone knows, patients could be dying in far greater numbers in hospitals in the beige regions than hospitals in the brown ones.”
In many ways, the article discussed above is almost all the way there. And with the resources and reputation of the New York Times, these reporters should have no problem getting sound opinions from outside experts that will allow them to focus their ideas a bit.
P.S. The Times should get up-to-date with their web-linking. For example, here’s a quiz for you. Find the problem in the paragraph below:
Wasteful spending — perhaps $700 billion a year — “does nothing to improve patient health but subjects you and me to tests and procedures that aren’t necessary and are potentially harmful,” the president’s budget director, Peter Orszag, wrote in a blog post characteristic of the administration’s argument.
Did you catch that? They quote from a blog but don’t link to it, instead linking to a generic search-for-Peter-Orszag page in the Times archive. I can’t imagine the editors were purposely avoiding the blog link; rather, they’re probably just not in the habit of linking to blogs. (I did a quick google and found Orszag’s blog here.)
P.P.S. Jonathan Skinner, a coauthor of the Dartmouth atlas, responds in detail on the New York Times website. The most impressive thing about this response is that Skinner published it on June 13, 2009–nearly a year before the Abelson and Harris article appeared. Now that’s what I call rapid response!
P.P.P.S. More here by Merrill Goozner, who talks about a number of ways in which it can be difficult to generalize from the Dartmouth study to make policy recommendations.
P.P.P.P.S. Still more here, from Hank Aaron at the Brookings Institution.