Responsible Software Engineering

Chapter 3. Incorporating Societal Context

This chapter was written in collaboration with Donald Martin Jr., who heads the Societal Context Understanding Tools and Solutions (SCOUTS) team in Google Research.

Technology changes how people behave. Thirty years ago, if we saw a person standing on a street corner all alone, arguing loudly and gesturing dramatically with both hands, we would have figured they were talking to themselves or even hallucinating. Today, we just assume they’re on the phone.

Likewise, people change how technology behaves. Build a system in-house and test it thoroughly, and it may seem to work fine. But then, when you release the system into our complex world, where it will interact with a vast assortment of people, new and surprising effects may emerge that no one foresaw.

Some of those effects have serious consequences too. The US health care industry, for example, commonly uses algorithms to predict the kinds of care a patient might need. One prominent algorithm in the 2010s was designed to reduce health care costs, and it was applied to about 200 million people per year, according to a 2019 paper in the journal Science. The algorithm computed a value, called a risk score, to determine whether a person had complex health needs that would benefit from more coordinated and targeted health care called a “high-risk care management program.” The algorithm’s designers faced a challenge, however. Most of their data came from health insurance claims, which described ...