AI for Humanity

CHAPTER 7Sustainable AI Governance

By far, the greatest danger of Artificial Intelligence is that people conclude too early that they understand it.

— Eliezer Yudkowsky

THE AI TRUST GAP

Risky Codes and Hidden Algorithms

In October 2021, Imane, a 44-year-old mother of three, faced interrogation in a small, non-descript office in Rotterdam.¹ She was not there because she had committed a crime. Instead, she was fighting to hang onto her welfare payments, a vital lifeline.

Imane and thousands like her in Rotterdam had been under surveillance by welfare fraud officers for years. Since 2017, the city has deployed a machine-learning algorithm to identify potential targets for probes. This system, trained on historical data, assigned each of Rotterdam's 30,000 welfare recipients a “risk score.”² Imane's life circumstances marked her as “high risk.”

This algorithm was Rotterdam's secret weapon in fighting bogus welfare claims. However, experts raised red flags, arguing that the algorithm perpetuated discrimination. Being female, young, a parent, non-Dutch speaking, or unemployed could spike one's risk score. Someone like Imane, a single mother immigrant from Morocco, was especially vulnerable to profiling.

In 2021, the city halted its use of the AI model after auditors determined that it lacked transparency and produced biased outcomes.

This pattern of deploying AI to combat fraud ...