Alexander at GiveWell writes:
The Disease Control Priorities in Developing Countries (DCP2), a major report funded by the Gates Foundation . . . provides an estimate of $3.41 per disability-adjusted life-year (DALY) for the cost-effectiveness of soil-transmitted-helminth (STH) treatment, implying that STH treatment is one of the most cost-effective interventions for global health. In investigating this figure, we have corresponded, over a period of months, with six scholars who had been directly or indirectly involved in the production of the estimate. Eventually, we were able to obtain the spreadsheet that was used to generate the $3.41/DALY estimate. That spreadsheet contains five separate errors that, when corrected, shift the estimated cost effectiveness of deworming from $3.41 to $326.43. [I think they mean to say $300 -- ed.] We came to this conclusion a year after learning that the DCP2’s published cost-effectiveness estimate for schistosomiasis treatment – another kind of deworming – contained a crucial typo: the published figure was $3.36-$6.92 per DALY, but the correct figure is $336-$692 per DALY.
Wow—a factor of 100! That’s pretty amazing. Things aren’t quite so bad, though:
We [the GiveWell team] do believe that the corrected DCP2 calculations are too harsh on deworming; our best estimate of the cost-effectiveness of deworming is in between the corrected and uncorrected DCP2 figures, at $30-$80 per DALY. In addition, there are strong arguments for deworming as an excellent intervention that do not depend on these figures. Overall we consider deworming a highly promising (though not the single most promising) intervention; we will be discussing our thoughts on this intervention further in the future.
Indeed, even $300 per DALY isn’t bad. I’d certainly pay $300 for a DALY myself.
Alexander continues:
More generally, we see this case as a general argument for expecting transparency, rather than taking recommendations on trust – no matter how pedigreed the people making the recommendations. Note that the DCP2 was published by the Disease Control Priorities Project, a joint enterprise of The World Bank, the National Institutes of Health, the World Health Organization, and the Population Reference Bureau, which was funded primarily by a $3.5 million grant from the Gates Foundation. The DCP2 chapter on helminth infections, which contains the $3.41/DALY estimate, has 18 authors, including many of the world’s foremost experts on soil-transmitted helminths.
I don’t know what a helminth is, but I’d certainly expect an expert on the topic to be within a factor of 100 of the correct value.
Here are some details:
When we [the Givewell team] examined the details of the official estimate, it struck us that nearly all of the DALYs saved (i.e., nearly all of the benefit) were coming from the reduction of a single symptom of a single worm infection: cognitive impairment due to ascariasis (we abbreviate this as CIDTA). Specifically, the figures going into the estimate implied that:
In a hypothetical population of 208,530 children (age 5-14 in Latin America) treated, 45,060 suffer from CIDTA. (Cells C44 and L44 in “ascariasis” sheet). That’s about 22%.
The disability weight of CIDTA is 0.463 (cell E8). While these figures are difficult to interpret, this implies that having CIDTA is about half as bad as being dead (disability weight 1.0), and only slightly less debilitating than being blind (disability weight 0.6). (See the official list of disability weights published alongside the DCP2.) These figures implied (to us) that CIDTA was not a matter of subtle cognitive impairment, but of mental handicap so severe as to truly prevent normal functioning.
The intervention in question – a single dose of albendazole – could completely restore normal mental functioning (i.e., completely eliminate disability associated with CIDTA) for one year.
These implications didn’t sync with the information we had from other sources, such as the Global Burden of Disease (GBD) report published alongside the DCP2. . . . the correct interpretation of Table 9 (for 5-14 year olds in Latin America) is that 45060 out of every million 5-14 year olds are at risk for CIDTA, and 5% of these actually have it – so 2253 out of every million 5-14 year olds have CIDTA. The official calculation assumes that in a population of 208,530 5-14 year olds, 45060 have CIDTA. The same types of errors apply to the other regions and conditions as well. . . .
P.S. This is pretty cool: GiveWell has a whole page detailing the mistakes they’ve made! That’s my kind of organization.
As a non-profit worker, agreed! Check out EWB Canada that does an annual failure report: http://www.ewb.ca/en/whoweare/accountable/failure.html
Here in Australia our government pharmaceutical subsidy scheme has an unofficial cost-effectiveness threshold of about $50K/QALY
[A QALY is not the same as DALY, but near enough if we are talking another two orders of magnitude]
Is this like hookworm in the U.S.? The Rockefeller Foundation’s campaign against hookworm in the U.S. South a century ago was hugely successful. It turned out a lot of people were a little impaired by hookworm, and the entire South seemed to have more on the ball after hookworm was brought under control.
There’s a RadioLab episode on “Parasites” that covers the Rockefeller Foundation campaign, and also some very speculative research that a few hookworms might be good for you. Very interesting, although the sources are of highly variable accuracy.
Andrew, thanks for posting this. This is how the system is supposed to work (the self-correcting part). A nice counterbalance to the necessary but depressing stories of misfeasance.
http://en.wikipedia.org/wiki/Helminth
What is a helminth? It is the correct zoological term for worms – specifically parasitic worms which resides inside the body. There are generally two types; Round worms like Ascaris, hookworms. The other type is the flatworm (such as tape worms (2 types: 1 for beef and one for pork). The official science is called Helminthology. The only organization I know about parasitic worm lovers is the Helminthological Society of Washington. I hope this clears up the question. Moral of the story: Be carefull where you put the decimal point.
[...] post, it came to my attention that GiveWell was also admitting failure when I read about it in a post on Andrew Gelman's [...]