97 Things About Ethics Everyone in Data Science Should Know

Chapter 70. Toward Algorithmic Humility

Marc Faddoul

Research Associate, School of Information, UC Berkeley

Defendant #3172 is an unmarried 22-year-old female. She previously served two months in prison for marijuana trafficking and has just been arrested for engaging in a violent public altercation with her partner. Is the defendant going to commit a violent crime in the three-month period before the trial? To answer such a question, many American jurisdictions use algorithmic systems known as pretrial risk assessment tools. Let’s consider one of the most common of these tools, the Public Safety Assessment (PSA).¹

When the PSA sees a high risk, it raises a red flag, and this automatically sends the defendant into detention, without further consideration from the judge to challenge the machine’s prediction. The stakes are high, as pretrial detention often comes with devastating consequences for the job and housing security of defendants, including those who are later proven innocent at trial. Tragically, 97% of these life-wrecking algorithmic red flags are in fact false alarms.² In other words, 3% of flagged defendants would have actually committed a violent crime had they been released, while the other 97% were detained unnecessarily. This is a strikingly poor performance, but it is somewhat unsurprising.

Foreseeing a crime in the near future is hard, and machines are not oracles. ...