Prosecutors' fallacy sampling and odds
To see why, suppose that police pick up a suspect and match his or her DNA to evidence collected at a crime scene. Suppose that the likelihood of a match, purely by chance, is only 1 in 10,000. Is this also the chance that they are innocent? It's easy to make this leap, but you shouldn't.
Here's why. Suppose the city in which the person lives has 500,000 adult inhabitants. Given the 1 in 10,000 likelihood of a random DNA match, you'd expect that about 50 people in the city would have DNA that also matches the sample. So the suspect is only 1 of 50 people who could have been at the crime scene. Based on the DNA evidence only, the person is almost certainly innocent, not certainly guilty.
This kind of error is so subtle that the untrained human mind doesn't deal with it very well, and worse yet, usually cannot even recognize its own inability to do so. Unfortunately, this leads to serious consequences, as the case of Lucia de Berk illustrates. Worse yet, our strong illusion of certainty in such matters can also lead to the systematic suppression of doubt, another shortcoming of the de Berk case.
I suspect that a bigger source of final mistakes than the "prosecutor's fallacy" is simply selective use of evidence. The strength of a piece of evidence depends not only on its likelihood, but also on the likelihood of all alternative evidence that prosecutors disregarded as unpersuasive. If the prosecution views the case from 10,000 angles, they will probably find a 1-in-10,000 fluke among them. Prosecutors don't have to fully explain the work they did to obtain incriminating evidence, so the jury may not be able to know its statistical value. That sounds like a central issue in the de Berk case.
-- Greg Kuperberg
more.