Jay Livingston points to an excellent rant from Peter Moskos, trashing a study about “food deserts” (which I kept reading as “food desserts”) in inner-city neighborhoods.
From the Times:
There is no relationship between the type of food being sold in a neighborhood and obesity among its children and adolescents.
Within a couple of miles of almost any urban neighborhood, “you can get basically any type of food,” said Roland Sturm of the RAND Corporation, lead author of one of the studies. “Maybe we should call it a food swamp rather than a desert,” he said.
Sure thing, Sturm. But I suspect you wouldn’t think certain neighborhoods are swamped with good food if you actually got out of your office and went to one of the neighborhoods. After all, what are going to believe: A nice data set or your lying eyes?
“Food outlet data … are classifıed using the North American Industry Classifıcation System (NAICS)” (p. 130). Assuming validity and reliability of NAICS occupational categories is quite a red flag. It means that if something is coded “445110,” then — poof — it’s a grocery store! What could make for easier analysis? But your 445110 may not be like my 445110. . . . A cigarette and lottery seller behind bullet-proof glass is not a purveyor of fine foodstuffs, and if your data doesn’t make that distinction, you need to do more than list it as a “limitation.” You need to stop and start over.
Here’s one way to do it: a fine 2010 Johns Hopkins study edited by Stephen Haering and Manuel Franco. They actually care about their data. Read the first page in particular for the problems of food-store categorization. It matters. And notice the sections titled “residents personal reflections on their local food environment” and “food store owners’ attitudes regarding stocking healthy food.” What a concept for researchers to actually talk to people!
Uh oh. I better be careful here, as I don’t talk to people myself! But I think that’s ok in my case; I’m just a political scientist . . .
I find this so frustrating because so much quantitative analysis is so predictably problematic, over and over, again and again, in exactly the same way. Here’s the mandatory (and then ignored) disclaimer (p. 134, emphasis added):
Possibly even more of a limitation is the quality of the … business listings, although this is a criticism that applies to all similar studies, including those reporting significant fındings…. More generally, categorizing food outlets by type tends to be insufficient to reflect the heterogeneity of outlets, and it is possible that more detailed measures, such as store inventories, ratings of food quality, and measuring shelf space, would be more predictive for health outcomes. Unfortunately, such data are very costly and time consuming to collect and may never exist on a national scale.
So let me get this right, because “all similar studies” use this flawed data, it’s OK? And because getting good data may be “very costly and time consuming to collect,” we’ll simply settle for what we have at hand? Bullshit!
You know, perhaps we never will have good data on a national level about what produce is sold in each and every store in America. I can live with that. But it is neither very costly nor time consuming to simply go into every store in any one neighborhood and see what is there. Do a spot check. Or at least read and learn from the John Hopkins study. I just found it on google without even trying. They managed just fine. And if a corner store sells three moldy heads of iceburg lettuce and some rotting root vegetables, it is not the same as Wholefoods simply because they’re both coded 445! . . .
And he keeps going from there. Connoisseurs of multilevel models will appreciate this bit from Moskos:
And if you have bad data, it doesn’t matter what fancy quantitative methods you use. It’s putting lipstick on the damn pig of correlation. Garbage in, garbage out:
The primary dependent variables (i.e., counts of food consumption) are regressed on the explanatory variables using negative binomial regression models, a generalization of Poisson models that avoids the Poisson restriction on the mean-variance equality.
Wow! Negative binomial Poisson regression models to avoid the mean-variance equality restriction. I (to my shame) no longer have any idea what that means, even though Poisson regressions were all the rage when I was in graduate-school. But I do remember the fatal flaw of non-random missing data.
I’m not against quantitative methods. I’m against bad research.
Yup. To put it another way, the researchers should just use the damn overdispersed Poisson regression and don’t make such a big deal about it. The quoted paragraph is a paragraph that wasn’t needed. You should never be doing a non-overdispersed Poisson regression anyway; it shouldn’t even be an option. Mean-variance equality, indeed.
P.S. Moskos also wrote a book called In Defense of Flogging which, as you might imagine from the title, recommends flogging or caning as an alternative to prison as a punishment for convicted criminals. I’m glad somebody wrote this book. I’ve thought for a long time that the flogging alternative is a good one, but when I mentioned it to my friends who were law professors, they said it would never fly. I agree with Moskos, who writes:
After all, who hasn’t committed a crime? Perhaps you’ve taken illegal drugs. Maybe you once got into a fight with a friend, stranger, or lover. Or you drove back from a bar drunk. Or you clicked on an online picture of somebody who turned out to be a bit young. Maybe you’re outdoorsy and were caught hunting without a permit.
Or maybe you’re a boss who knowingly hired illegal immigrants. Perhaps you accepted a “gift” from a family member and told the IRS it was a loan. Or did you go for the white-collar big leagues and embezzle millions of dollars? In truth, you may be committing some crimes you don’t even know about. If your luck runs out, you can end up in jail for almost anything, big or small. And even if you’re convinced that you’re the most straitlaced, law-abiding person in the world, imagine that through some horrific twist of fate, you were accused of a crime. It’s not inconceivable; it happens all the time.