Aurelian Muntean writes:
I have read an article on NPR and the journal article that spun this news.
What draw my attention was the discussion in terms of causation implied by one of the authors of the article interviewed in the NPR news, and also by the conclusions of the article itself claiming large effects.
Although the total sample (self-selecting pregnant women) seems very large (85,176) the subsamples (270 out of which 114 were in the sub-subsample revealing statistically significant association) used to support the analysis seem to be too small. Or not?
My response:
The different sources of information do seem to be in some conflict:
– The JAMA article reports the autism rate of 1 per 1000 for children of mothers taking folic acid and 2 per 1000 for children of mothers not taking folic acid. (They also report the adjusted odds ratio as 0.6 rather than 0.5, indicted that the two groups differ a bit in some background variables.)
– The NPR article has this quote: “‘That’s a huge effect,’ says Ian Lipkin, one of the study’s authors . . . ‘when you start talking about autism, a disorder that has an incidence of 1 percent or higher, that really does bring it to home,’ Lipkin says. ‘That is a substantial risk.'” How do you get from 1 or 2 per 1000 to “1 percent or higher”?
Another issue that arises is multiple comparisons: they studied at least three subpopulations (“114 with autistic disorder, 56 with Asperger syndrome, and 100 with PDD-NOS”) and at least two predictors (“Similar analyses for prenatal fish oil supplements showed no such association with autistic disorder”). So it seems like we’re seeing the most statistically significant of at least 6 comparisons.
In general I recommend addressing multiple comparisons problems by using hierarchical models, but it’s not clear to me exactly what I would do in this case. It would be good to have a general method to recommend for this sort of problem. I think it would involve regularization and informative priors.
Finally, there’s causality—do the folate and non-folate parents differ, on average, in important ways not controlled for in the analysis? I have no idea. I will say, though, that we followed the folate recommendation ourselves.
I believe Ian Lipkin is referring to the oft reported autism spectrum disorder population rate from a CDC study when he says “1 percent or higher.” But of course, there is still a mismatch there.
I note these lines from the study:
“The use of other vitamins and minerals in addition to folic acid did not appear to affect risk of autistic disorder. The analyses based on the food frequency questionnaire data from week 22 did not reveal any apparent association between maternal total daily folate intake in week 22 (diet and supplements combined) and subsequent risk of autistic disorder in children. The analysis in which cases were stratified according to language level suggested that the inverse association may be strong in children with severe language delay and weak in those with moderate or no delay. The analysis stratified by year of birth suggested that the inverse association may be stronger in the older children (born in 2002-2004) than in the younger children (born in 2005-2008).”
Kind of complicated, especially when you think about the lack of relation to week 22 reported folate intake. I don’t know how you could pul these factors apart – and the others, including where each diagnosed child is on the spectrum*, etc. – without some more levers, meaning understanding that generates ways of formulating testable questions. The overall relationship is a knotted ball. I suppose you do more work, treat the knotted ball as a whole and try to figure out over time what may be affecting what.
*Which if it’s in there, I missed, which isn’t uncommon since I skim back and forth – I hash – things like this.
Wait isn’t “stratifying according to language level” what we’ve been calling “controlling for post-treatment variables” in this blog, I mean, you treat with folate, and then years later either you have some form of autism in which presumably language would be delayed, or you don’t, but you wouldn’t want to be looking for interactions of folate with post-treatment language levels. It’d be a different story if you were treating the children with folate after classifying their language level, but that’s not what’s going on.
When it comes to diet and public health I’m at a stage where I only trust recomendations in The Onion.
Publication bias + cut throat competition for grants to pay 75% of your salary + relentless pressure to publish 10 or more items per year = [your choice of adjective] science.
And perhaps a minimal or even negative return on investment for efforts to get less wrong as this invariably requires an increase in reported uncertainty and unavoidable decline of value of the (potential) work in the eyes of those with money to fund.
The likely hope being those with money to fund may realize/get tired of being repeatedly mislead.
That assumes those doing the funding care about being misled; as opposed to being confirmed in their beliefs, or shown to be funding headline grabbing results.
After all, what funder wants to be told good science is a very pedestrian affair. One that proceeds one small step at a time.
Not assuming those doing the funding care about being misled but that it may be possible to change that.
Yes, I agree. The problem is the status quo is an equilibrium. We need to find some exogenous shock to change it.
But in some ways the problem may bear the seed of its own solution. With a glut of PhDs, the spread of MOOCS, and so on there is a huge mismatch between supply and demand in academia. One possibility is this ramps up all the aforementioned problems to the point reform becomes inevitable. I think we are beginning to see this happen.
John is correct, the 1% or higher refers to the official CDC autism rate. It’s been changing pretty rapidly in the last couple of years. form 1 in 150, to 1 in 88, (to 1 in 55?).
The multiple comparisons issue that you bring up is actually not a problem with multiple comparisons, but with the changes in how things are diagnosed over the past 10 years. Aspergers/Autism/PDD-NOS all exhibit the same complex of symptoms, and this is one reason that the new DSM changes the diagonstic criteria and eliminates the aspergers diagnosis. (If you google this, you will see quite a controversy surrounding this change, a lot of “Aspies” wanted to keep their separate label.) Confounding all of these changing criteria is that sometimes parents aren’t ready/willing to accept an autism diagnosis, and accept the “Pervasive Developmental Delay-Not Otherwise Specified” much more easily. If you tried to treat these as three separate groups, you could wind up with a huge mess, and some serious construct validity questions. The changing definition is one reason (among many) for why the CDC rate keeps changing. Treating them as one “Autism Spectrum Disorder” for the purposes of the analysis was almost certainly the right choice.
The 0.1% vs. 1% really makes me worry more about some sort of problem with the sample. The methods for finding the ASD kids in the sample seem unlikely to have found all of the ASD kids. Moreover, if you have a 10 year old kid without a diagnosis, then they probably really don’t have an ASD (likely true negative). But the diagnosis methods for 3 year olds aren’t very reliable (false negatives?). And a lot of kids with ASD still aren’t even referred for screening until they have problems in kindergarten (more false negatives). The fact that they didn’t seem to deal with these sampling/sensitivity issues is where I’d worry, before even thinking about the analysis.