David Hogg pointed me to this news article by Angela Saini:
It’s not often that the quiet world of mathematics is rocked by a murder case. But last summer saw a trial that sent academics into a tailspin, and has since swollen into a fevered clash between science and the law.
At its heart, this is a story about chance. And it begins with a convicted killer, “T”, who took his case to the court of appeal in 2010. Among the evidence against him was a shoeprint from a pair of Nike trainers, which seemed to match a pair found at his home. While appeals often unmask shaky evidence, this was different. This time, a mathematical formula was thrown out of court. The footwear expert made what the judge believed were poor calculations about the likelihood of the match, compounded by a bad explanation of how he reached his opinion. The conviction was quashed. . . .
“The impact will be quite shattering,” says Professor Norman Fenton, a mathematician at Queen Mary, University of London. In the last four years he has been an expert witness in six cases . . . He claims that the decision in the shoeprint case threatens to damage trials now coming to court because experts like him can no longer use the maths they need.
Specifically, he means a statistical tool called Bayes’ theorem. . . .
In the shoeprint murder case, for example, it meant figuring out the chance that the print at the crime scene came from the same pair of Nike trainers as those found at the suspect’s house, given how common those kinds of shoes are, the size of the shoe, how the sole had been worn down and any damage to it. Between 1996 and 2006, for example, Nike distributed 786,000 pairs of trainers. This might suggest a match doesn’t mean very much. But if you take into account that there are 1,200 different sole patterns of Nike trainers and around 42 million pairs of sports shoes sold every year, a matching pair becomes more significant.
The data needed to run these kinds of calculations, though, isn’t always available. And this is where the expert in this case came under fire. The judge complained that he couldn’t say exactly how many of one particular type of Nike trainer there are in the country. National sales figures for sports shoes are just rough estimates.
And so he decided that Bayes’ theorem shouldn’t again be used unless the underlying statistics are “firm”.
I don’t know enough about the case to have any sense of whether I agree with the judge. It seems reasonable to require that numbers be “firm” if they are to be used in a court case. But, what does the judge recommend doing if the numbers aren’t firm? Just guessing? Just using likelihoods and no base rates? What if the base rates are firm but the likelihoods are not?
P.S. Cosma Shalizi writes:
In so far as the judge said that Bayes’s rule “shouldn’t … be used unless the underlying statistics are ‘firm’,” he was being entirely reasonable.
I agree—as long as the judge recognized the problems with using any statistical method when the underlying statistics are not “firm.” The garbage-in, garbage-out problem is not unique to Bayes!
Sounds like a place where a “worst case” analysis would be appropriate.
I think this may be the beginning of a beautiful employment relationship between many statisticians and courts. All statisticians should rejoice at the nit-picking accuracy required. Who knows, perhaps set theorists will be called to the stand soon.
With regard to “firmness”, I think (and so do you – correct me if needed), we just need a more elaborate model to make its initial assumption “firmer” (i.e. less controversial). In this case we could have a probabilistic model of sales numbers, probabilistic models of shoe degradation and so on. Finally the initial assumption could look like: the sales numbers cannot be off by more than a factor of 10. And the final verdict would be: Under these assumption, such and such would occur by chance less than 1 in a 100,000.
Cosma Shalizi commented on the issue in his blog. But I think he was too harsh in his critique of the Bayesian approach over there…
National sales figures seem irrelevant in any case. What really matters is how common such shoes are amongst local people who might plausibly have been the murderer.
However, asking for perfection seems too much. Once the expert witness has presented a calculation based on what seem like realistic assumptions, looking at things in as much detail as it seems reasonable to do, the onus ought to be on the defense to rebut this, by, for instance, presenting evidence that there was a fad for that exact type of shoe amongst the accused, the victim, and their acquaintances, so that actually all the plausible suspects wear that exact type.
This is surely how things work for other types of evidence, else there would be no need to allow for any defense witnesses – the defense would just point out that the prosecution’s case didn’t amount to an absolute proof, and that would be that.