Using propensity scores to estimate the effects of seeing gun violence

Jeff Fagan forwarded this article on gun violence by Jeffrey Bingenheimer, Robert Brennan, and Felton Earls. The research looks at children in Chicago who were exposed to gun violence, and uses propensity score matching to find a similar group who were unexposed. Their key finding: “Results indicate that exposure to firearm violence approximately doubles the probability that an adolescent will perpetrate serious violence over the subsequent 2 years.”

I’ll first give a news report summarizing the article, then my preliminary thoughts.

Here’s the summary:

Controversial Study Suggests Seeing Gun Violence Promotes It

Constance Holden

A longitudinal study of Chicago adolescents has concluded that even a single exposure to firearm violence doubles the chance that a young person will later engage in violent behavior. The study may once again stoke up the debate over juvenile violence; it has already triggered criticism over the unusual statistical method it employs.

The work is part of the decade-old Project on Human Development in Chicago Neighborhoods, run by Harvard University psychiatrist Felton J. Earls. On page 1323, Earls and two health statisticians describe how they used a relatively new technique called “propensity score stratification” to create, through statistical means, a randomized experiment on propensity toward violence from observational data.

Over a 5-year period, the researchers conducted three interviews with more than 1000 adolescents initially aged 12 to 15. In the first, they gathered extensive data on variables such as family structure, temperament, IQ, and previous exposure to violence. Halfway through the study, the subjects were asked if, in the prior 12 months, they had been exposed to firearm violence–defined as being shot or shot at or seeing someone else shot or shot at. Then at the end of the period, the 984 subjects remaining were asked if they had engaged in any violence–defined as participation in a fight in which anyone got hurt as well as firearm-related incidents, including carrying a gun.

Figure 1Violence debate. A study of Chicago adolescents indicates that seeing a murder may lead to later gun violence by the observer.

“If you just compare exposed and unexposed, the exposed were three or four times as likely to be [violence] perpetrators,” says lead author Jeffrey B. Bingenheimer, a Ph.D. candidate at the University of Michigan School of Public Health in Ann Arbor.

The authors then went to great lengths to weed out confounding factors. Subjects were ranked according to “propensity” scores: a cumulative tally of 153 risk factors that estimated the probability of exposure to gun violence. They were then divided up according to whether or not they had reported such exposure and whether or not they had subsequently engaged in violent behavior. Those with the same propensity scores but different exposures were compared with each other. In this way, the authors claim, they controlled for a host of individual, family, peer, and neighborhood variables.

Even with this analysis, exposure to gun violence predicted a doubling of the risk for violent behavior–from 9% for unexposed to 18% among the subjects who reported exposure, says Bingenheimer. And it didn’t take repeated exposures–“the vast majority” of subjects reported only one, he says. Can a single experience of seeing someone shoot at someone else make an individual more violence-prone? “That doesn’t seem improbable to me,” says Bingenheimer. “It could be for only a minority, but a very large effect for that minority.”

Developmental psychologist Jeanne Brooks-Gunn of Columbia University, one of the scientific directors of the Chicago neighborhoods project, agrees that a single exposure might have a profound effect, even on a hitherto nonviolent individual. “Nobody’s done this kind of analysis before,” she says, and nobody has focused just on gun violence, which “clearly is a very extreme type of violence.”

But a number of other scholars have deep misgivings about both the study findings and the methodology. Psychiatrist Richard Tremblay of the University of Montreal in Canada says the study does not demonstrate that “those who are nonviolent to begin with will become violent.” Indeed, the authors didn’t address this point directly because a lack of subjects in the lowest-risk category led them to eliminate it from their analysis.

Because the remaining subjects already had some violence risk factors, the results don’t surprise Tremblay. He compares the work to looking at whether alcoholics are more likely to drink if they are exposed to alcohol. It is already well known, he says, that “if individuals at a high risk of violence are in an environment with violence, they’re more likely to be violent.”

Economist Steven Durlauf of the University of Wisconsin, Madison, calls the study an “implausible modeling of violence exposure.” The authors assume that two individuals with the same propensity rankings are equally likely to encounter violence, he says. But such exposure may not be random; rather, it probably stems from “something that has not been measured”–such as recklessness, says Durlauf. Nobel Prize-winning economist James Heckman of the University of Chicago agrees, calling the study “potentially very misleading.” Adds Heckman: “This is why this kind of statistics is not science. This is why you find out orange juice causes lung cancer one week and cures it the next.”

But Brooks-Gunn defends the innovative study. The propensity scoring technique “comes the closest we have to any experiment, which is why I think the results are so strong,” she says.

My thoughts:

I see merit in the arguments of both sides. I don’t know the context of Heckman’s particular comment about orange juice, but perhaps the issue is that it contributes to cancer for some people, under some conditions, while reducing the risk of cancer for other people, under other conditions. Matching methods work by restricting analysis to a subset of the population that is well-matched for the treated and control people. So any summary of such an analysis should consider treatment interactions–that is, ways in which treatment effects can vary given pre-treatment predictors.

However, I don’t think it’s right to simply dismiss this sort of observational study. For one thing, people will be making these comparisons anyway, and comparing matched groups should be better than simple unmatched comparisons.

3 thoughts on “Using propensity scores to estimate the effects of seeing gun violence

  1. I have a couple of comments:

    1. What about this whole correlation is not the same as causation problem? Here, it strikes me as possible that that the propensity scores are measuring the propensity to enter a gun culture. This, of course, is not a criticism of propensity scores per se.

    2. Was any model checking done on the propensity scores? Some of the variables used in their creation look like they should have an effect on gun use, but if the model for the propensity score doesn't fit well, one could imagine the discrepency between the "true" propensity and the estimated propensity explaining the results.

    Bob

  2. I have to say that my gut feeling is that propensity-score modelling, whether or not it's providing a valid instrument, isn't really modelling the underlying reality here.

    When someone shoots someone, it's usually for a reason and that reason isn't anything like "because a random draw from a normal variate exceeded the individual-specific propensity score". It's a historic event with a whole series of causes and an analysis which doesn't bring reasons into it is not really going to be reliable.

    I mean, would you try to explain "My name is Inigo Montoya. You killed my father. Prepare to die!" in terms of exposure and propensity?

  3. Bob,

    Regarding your point 1: yes, that's clearly an issue: possible "lurking variables." The researchers do control for pre-exposure "temperament" measurement, which of course isn't perfect, but seems like a reasonable start. Or, to start from the other end, the comparison of raw data doesn't seem quite right, and I would think the adjustment takes things in the right direction. The statement that "for these two groups of kids, matched on all these pretreatment measurements, exposure to firearm violence is highly correlated to later perpetrating serious violence" is interesting, even if one can never quite close the door on lurking-variable-type explanations.

    Regarding your point 2: this is a tricky issue. I've heard it said that misspecification of the propensity score model is not such a problem; see the paper by Guido Imbens discussed here, but these issues still confuse me (which is why we held this meeting).

    Dsquared,

    I don't see any contradiction between your statement that "when someone shoots someone, it's usually for a reason…" and the model's claim that a particular exposure (e.g., seeing gun violence) could increase the probability of a violent act. It's just a matter of changing your threshold for how much of "a reason" is necessary to trigger the act. I don't see the issue of "historic event" being relevant. Whether or not something has a whole series of causes, it is possible that a treatment can affect the probability of it happening.

    In the context of your example: yes, if Inigo was exposed at a young age to violent swordplay (not to mention, actually being trained to fight) then, yes, that could make it more likely for him to resort to violence. Perhaps if he had been raised as a Buddhist, he'd just calmly wait for his father's killer to meet divine justice, or whatever S. Morgenstern might say in this case.

Comments are closed.