Responsible Software Engineering

Chapter 2. Creating AI Systems That Work Well for Everyone

Today’s AI models can analyze images and describe their contents with uncanny accuracy, but they can also exhibit bias. Suppose you fire up one of these models and show it a picture of a man in a white lab coat with a stethoscope around his neck (as on the left side of Figure 2-1).¹

Figure 2-1. Pictures of a generic man and woman with white lab coats and stethoscopes

The model responds, “This is a doctor.” Next, you show the AI model a picture of a woman in a white lab coat with a stethoscope. The model responds, “This is a nurse.” The AI model has displayed a bias that doctors are male and nurses are female, which is a stereotype in various cultures.

I’ve also seen AI models identify the picture on the right as “a female doctor” and the picture on the left as just “a doctor.” The addition of female, known as marked language, is another way in which software applications can display subtle, unwanted bias that favors one group over another.

If my example seems harmless, consider what could happen if the same AI model were the engine behind a website for serving job ads. What if it pitched doctor positions to male users more often than female users? Now we’re talking about automated job discrimination that can lead to ...