97 Things About Ethics Everyone in Data Science Should Know

Chapter 39. Algorithms Are Used Differently than Human Decision Makers

Rachel Thomas

Cofounder, fast.ai; Director, USF Center for Applied Data Ethics

People often discuss algorithms as though they are plug-and-play, interchangeable with human decision makers—just comparing error rates, for instance, when deciding whether to replace a human decision maker with an algorithmic result. However, in practice, algorithms and human decision makers are used differently, and failure to address those differences can lead to a number of ethical risks and harms.

Here are a few common ways that algorithms and human decision makers are used differently in practice:

• Algorithms are more likely to be implemented with no recourse process in place.

• Algorithms are often used at scale.

• Algorithmic systems are cheap.

• People are more likely to assume algorithms are objective or error-free.

There is a lot of overlap between these factors. If the main motivation for implementing an algorithm is cost cutting, then adding an appeals process (or even diligently checking for errors) may be considered an “unnecessary” expense.

Consider one case study: after the state of Arkansas implemented software to determine people’s health care benefits, many people saw a drastic reduction in the amount of care they received but were given no explanation and no way to appeal. Tammy Dobbs, a woman with cerebral palsy ...