Machine Learning for High-Risk Applications

Chapter 4. Managing Bias in Machine Learning

Managing the harmful effects of bias in machine learning systems is about so much more than data, code, and models. Our model’s average performance quality—the main way data scientists are taught to evaluate the goodness of a model—has little to do with whether it’s causing real-world bias harms. A perfectly accurate model can cause bias harms. Worse, all ML systems exhibit some level of bias, bias incidents appear to be some of the most common AI incidents (see Figure 4-1), bias in business processes often entails legal liability, and bias in ML models hurts people in the real world.

Figure 4-1. The frequency of different types of AI incidents based on a qualitative analysis of 169 publicly reported incidents between 1988 and February 1, 2021 (figure courtesy of BNH.AI)

This chapter will put forward approaches for detecting and mitigating bias in a sociotechnical fashion, at least to the best of our ability as practicing technicians. That means we’ll try to understand how ML system bias exists in its broader societal context. Why? All ML systems are sociotechnical. We know this might be hard to believe at first, so let’s think through one example. Let’s consider a model used to predict sensor failure for an Internet of Things (IoT) application, using only information from other automated sensors. That model would likely have been ...