Predicting death sentences–is predictability an indicator of aribitrariness?–and what does it mean for a statistical method to be called a “computer system”?

Andrew Sullivan links to this news article which links to this research article by Stamos Karamouzis and Dee Wood Harper called “An Artificial Intelligence System Suggests Arbitrariness of Death Penalty”:

The arguments against the death penalty in the United States have centered on due process and fairness. Since the death penalty is so rarely rendered and subsequently applied, it appears on the surface to be arbitrary. Considering the potential utility of determining whether or not a death row inmate is actually executed along with the promising behavior of Artificial Neural Networks (ANNs) as classifiers led us into the development, training, and testing of an ANN as a tool for predicting death penalty outcomes. For our ANN we reconstructed the profiles of 1,366 death row inmates by utilizing variables that are independent of the substantive characteristics of the crime for which they have been convicted. The ANN’s successful performance in predicting executions has serious implications concerning the fairness of the justice system.

I don’t really see why the predictability of death sentences–in their data set, they say they “successfully classified 147 out of 158 non-executed inmates (93.0%) and 130 out of 142 executed inmates (91.5%)”–is evidence that death sentencing is “arbitrary.” Predictable seems like the opposite of arbitrary.

We do know, though, that most death sentences get reversed:

We collected data on the appeals process for all death sentences in U.S. states between 1973 and 1995. The reversal rate was high, with an estimated chance of at least two-thirds that any death sentence would be overturned by a state or federal appeals court. Multilevel regression models fit to the data by state and year indicate that high reversal rates are strongly associated with higher death-sentencing rates and lower rates of apprehending and imprisoning violent offenders. In light of our empirical findings, we discuss potential remedies including “streamlining” the appeals process and restricting the death penalty to the “worst of the worst” offenders.

P.S. To make a more parochial comment, I’m surprised the news article was headlined “Computer predicts . . . ” as if somehow this was done by HAL rather than a human statistical analyst. I don’t know how the neural network method of Karamouzis and Harper compares to BART or logistic regression–I’m willing to believe it’s better–but it seems a little funny to me to refer to it as a “computer system” rather than a “prediction method” or a “statistical method” or a “prediction algorithm.”

14 thoughts on “Predicting death sentences–is predictability an indicator of aribitrariness?–and what does it mean for a statistical method to be called a “computer system”?

  1. I would think that using variables independent of the crimes committed does show, or point to arbitrariness, as the crime itself should be the main (actually, only) predictor of the sentencing.

    Say they use income, race, age and gender to predict the outcome of the sentencing (I would guess they're useing something like this) than the outcomes should be mostly unpredictible, shouldn't they? The system shouldn't hedge on these variables at all.

    This would be an easy (if time consuming) logistic regression problem I would suspect.

    Now, I suppose these variables may be predictors of the heinousness of the crimes involved. If that's the case, well, there's another grant in there somewhere.

    I would suspect demographics do not predict heinousness of crimes. Most of the evidence I think already shows this.

  2. I think the arbitrariness lies in the fact that they are predicting based on: "variables that are independent of the substantive characteristics of the crime for which they have been convicted"

    so I assume this means things like age, sex, race, weight, height, income etc. If the application of the death penalty primarily has to do with how much of a thug you look like independent of the crime you committed, then that's pretty "arbitrary".

  3. My sense is that they're using the word arbitrary in a different sense than you are. They have 19 variables per inmate input into the neural network, and their claim is that these variables (e.g., race, state, year of arrest, inmate identification number) should not be predictive in a fair judicial system, and yet they are. Therefore the system is more arbitrary and less fair.

    Three of their variables are: third most serious capital offence, second most serious capital offence, and first most serious capital offence, and those would seem to relevant, unless I'm misunderstanding what they mean.

  4. The point of the predictability -> arbitrariness inference is not predictability alone, but rather predictability using only factors "independent of the substantive characteristics of the crime".

    If an execution or stay can be predicted without reference to characteristics the crime, then something is very broken. On the other hand, if you couldn't predict the outcome without the characteristics of the crime were both necessary and sufficient for building a predictor, then we could say that something makes sense about the process (whether we agree or not with the death penalty itself).

  5. I think there might be a disconnect between 'fair' and 'arbitrary'.

    Their reasons seems to be: "However, given that the variables employed in the study, have no direct bearing on the judicial process raises serious questions concerning the fairness of the justice system."

    There were several variables to me that seemed important to the judicial process though:
    # Third most serious capital offence
    # Second most serious capital offence
    # First most serious capital offence
    # Legal status at time of capital offense
    # Prior felony conviction(s)

    But I didn't see any data on how influential these were in the decision process of the neural net. However, if sex, hispanic origin, or heck, even inmate identification were significant in the decision process then I would agree that the process is unfair (but not arbitrary).

  6. I just read this article and take back most of what I said about this being an interesting argument. If the data analysis had been done well, it might be interesting, but the data analysis in this article is clearly not done very well.

    See my own blog for my detailed observations.

  7. I had the same thought above the fold that you expressed below it: predictability is not compatible with arbitrariness. But I think arbitrary in this context refers to judicial arbitrariness, i.e., "[predictable using] variables that are independent of the substantive characteristics of the crime…"

    Knowing nothing more than this, I'd bet dollars that victim and perpetrator race and their interaction are predictive.

  8. "Predictable seems like the opposite of arbitrary"

    Not if you interpret arbitrariness as lacking good reason (rather than lacking any reason at all). If convictions were perfectly predicted by the race of the accused, wouldn't that suggest that convictions were arbitrary – in the sense that they're related to something they shouldn't be related to (and by implication, not related to things they should be related to)?

  9. I think they mean arbitrary in the sense that the decision to apply the death penalty can be explained by an illogical criteria, not that it's random.

    "[V]ariables that are independent of the substantive characteristics of the crime" strikes me as the polite way of saying "illogical criteria".

    Since journalists type articles into computers I suppose news articles are written by 'computer systems' as well.

  10. "Arbitrary" seems to be used as a term related to normative judgments of fairness, not as a statistical or causal claim.

    But I'm surprised that the issue of the quality of the analysis was raised only once on a statistical modeling blog.

    Wouldn't it be the case that some variables they use are correlated with potential causes for committing crimes punishable by death?

    If so, then why would you say that the predictability of the execution implies the unfairness of the system?

  11. One question I haven't seen addressed (I skimmed the article but could easily have missed something):

    isn't this whole thing bogus if the variables they're using ALSO predict 'relevant' information such as seriousness of crime etc etc?

    the data are available (for people from ICPSR institutions) at

    http://www.icpsr.umich.edu/cgi-bin/bob/newark?stu

    (this answers one of the criticisms in Ted Dunning's blog entry)

    Ben Bolker

  12. Re: Bolker

    My linguistics prof taught me to be very suspicious of perception.

    Would it be that these variables predict severity or would it be that these variables predict the perception of the severity. I say the latter would be the stronger relationship. Which is not entirely inconsistant with the original work. It is the perception of severity that creates the sentencing and not the underlying unobservable variable of actual severity.

Comments are closed.