Chapter 1. Fairness, Technology, and the Real World

What is fairness? Everyone has a slightly different definition. Many of the great divides in society result from differing ideas about fairness. It’s an age-old debate.

In this book, I won’t delve into the philosophy or social histories of how fairness has been defined over time and geography. Rather, I’ll take a practical perspective on the matter. Practical considerations most often come up in the form of three fundamental questions a society needs to answer in order to function:

Who gets what? (Rules of allocation)
How do we decide who gets what? (Rules of decision)
Who decides who decides? (Rules of political authority)¹

Some of the social and philosophical divides in the world originate from these basic questions. Now there’s a new and interesting wrinkle in the age of algorithms—one that is not sufficiently acknowledged. We are still asking and answering these same questions, but now algorithms are part of that process.

Software engineers and UX designers don’t usually ask these questions at team meetings. And yet they make such determinations every day in their work. The downstream effects of their products have social ramifications that affect who gets what and why.

This book provides both conceptual tools and coding examples to address fairness questions from the point of view of writing computer code and designing digital products. Much of the content emphasizes machine learning, but I also discuss digital products more widely. As I address ML and other digital products throughout this book, I’ll try to emphasize the following fundamental, pragmatic, and unavoidable questions you will need to answer if you work on digital products, even if you don’t recognize the choices you are making:

Is it fairer for everyone to have the same opportunities or to have the same outcomes? Equality of opportunity or equality of outcome?
Is it fairer for decisions to be uniform or to embody an element of human empathy? Impartial justice or individual allowances?
Is it fairer to let people know how decisions are made or to have an opaque system to prevent cheating? Transparency or security?

Such questions are about trade-offs, some necessary and inevitable, and others possibly solvable. We also have more specific questions about implementations, details, and human response:

Is it a problem when a machine learning model has different average predictions for different genders? Does it matter how different the average predictions are? Does it matter whether the model is used in high-stakes or low-stakes decisions?
What kind of metadata about user behavior is it ethical to collect from apps? When is metadata collection justified for customization, and when is it a form of spying?
Will people praise a credit-rating algorithm for providing uniformity or resent it for failing to see their individual circumstances? Does any online application for credit need a free-form box for the applicant to supply necessary information or context?

Technological, procedural, and institutional tools are all needed to pose these questions and develop appropriate mechanisms to answer the questions and implement the chosen policies moving forward. For example, from a technological perspective, you need a data science or ML pipeline with sufficient documentation and accessibility of relevant attributes, such as the internal specs of a model and information about membership in protected categories for individuals included in your data set. From a procedural standpoint, you need training or careful consultation regarding the process for making these decisions in a way that reflects appropriate fairness norms, logic, and ethical consistency with your organization and your society’s background laws and cultural codes of ethics. From an institutional standpoint, you need leaders and managers of an organization who set the right tone to keep fairness issues on the radar.

My goal is to point out how fairness questions come into play when building digital systems, particularly systems powered by contemporary machine learning methodologies and other data-science-driven insights. I’ll call all of these ML for shorthand; that is, I don’t distinguish deep learning from analytics from machine learning from statistical analysis, etc.

I take a broad view of how interfaces and code written by interface designers, data scientists, machine learning engineers, and others can violate important fairness norms. With this goal, I discuss best practices and technical tests that can be used in pursuit of a more just digital world by people designing digital products and writing data-driven code.

We’ll think about the interfaces that wrap our products in terms of how people see them rather than just the code that powers them. We’ll think about how the human readable aspects of our products have ramifications for the humans that read and use our products, and also the humans who don’t.

Fairness is a hot topic. However, concerns about fairness in technology are far from new. What’s more, the concerns raised recently in popular discourse about machine learning, automation, and all things digital are not as new as the media hype implies.

Fairness in Engineering Is an Old Problem

Technology is neither good nor bad; nor is it neutral.

Melvin Kranzberg

New technologies and their downstream social impacts have always had fairness implications, and many of these effects center on the same questions that vex us now. Earlier technologies need not look much like the information technologies driving the last several decades of innovation to be relevant. What matters is the social element and the embedding of the technology in a sociotechnical system. This fundamental connection between the social and the technical is a constant across time and technologies. It could be a wheel, a railroad engine, a vacuum cleaner, or a computer. Any of these could affect social ordering and organization, and all of them did.

Our societies like to tell a story of “technological progress = better life = fairer and better human experience.” However, just about every era of invention, and every individual invention, has easily identifiable victims.

For example, you could make the argument that the current unequal distribution of global wealth is very much connected to Western Europe’s Renaissance and the associated advances in science and engineering. Guns! Advances in shipbuilding! Advances in navigation! All enabled colonialism. The inventors of these technologies could have avoided much of the resulting unfairness at the time (for example, de facto slavery in colonies), although whether they wanted to avoid such consequences is another question.²

Let’s consider another historical example. The next great technological period of Western culture, the Industrial Revolution, had fairness implications both within Western nations and internationally. This period, like the Renaissance, is often taught to Western schoolchildren as quite a good thing, with some pro forma nods to the resulting social chaos, dehumanization of work tasks, and increasing inequality that resulted.

What were some of the foreseeable unfair outcomes of the Industrial Revolution? Consider the way factory machinery was dependent on small bodies and fingers; children made the ideal factory employees for many dangerous tasks. The jump in society’s productivity was accomplished in part by the labor of children rather than by better opportunities for children.

Or consider that the style of human work most appropriate to factory tasks was monotonous and tedious, turning artisans and craftspeople into organic repetitive-motion machines. Was that good for these workers? Circumstances varied, but there were clearly distributional- and autonomy-related fairness considerations for people who lost discretion in their work and became machine-like factory workers. These outcomes were foreseeable, so the real question is whether the downsides were worth the upsides at the time, and to whom.

Similarly, when we write code or design technology interfaces now, we can sometimes foresee potential bad outcomes for certain identifiable stakeholders or third parties, and the question becomes whether the downsides are worth the potential upsides, and for whom?

I use the child labor example in part because that practice continued into the Gilded Age, a period of vast wealth and power disparity in the Western world organized around the control of new technologies and the downstream demands created by those technologies (railroads, manufacturing, fuel). The design of manufacturing technologies and the sociotechnical complex that complemented the technologies made for a very unfair situation. Children in lower-class families who found themselves working as human cogs in factory machines had little chance to escape this grind; being in it had nothing to do with merit or hard work and everything to do with being born at the wrong place and in the wrong time. Meanwhile, as the children suffered a lack of education and even basic safety, their labor generated enormous amounts of wealth that were concentrated in the hands of very few. Importantly, market pressures were not enough to stop child labor. Rather, it took decades of political crusading before strong new laws were passed to end factory use of child labor in Western countries. We learn a historical lesson here because, likewise, many people likewise believe that something beyond market pressures will be necessary to make ML fair.

These are just a few examples to show that technology does not necessarily self-regulate, via either market or social pressures. We should keep this in mind when considering novel legal proposals to address the new generation of concerns about unfairness, such as algorithmic discrimination, invasion of privacy, and the rise of surveillance capitalism. Indeed, many large tech companies have even gone on record indicating that private solutions are either unlikely to materialize³ or not enough without government assistance in coordination and enforcement.⁴ Fairness problems arising from technology are an old problem, and they are not always solved by the market or by social norms. As of this writing in late 2020, popular and industry opinion alike seem strongly in support of fairness interventions from lawmakers.

Our Fairness Problems Now

Many say we are now in a second Gilded Age, as wealth and income disparity are again on the rise worldwide. While we face increasing wealth disparities, we live in a hypertechnological era, and centers of tech reflect a dramatic trend toward income inequality. This is not to argue that tech causes inequality but to point out that our current advances in technology seem correlated with increasing inequality. In some ways, we see the old ills reproduced with a newer technology center. But we also have problems associated with qualitatively new ways of doing business or making money thanks to the big data revolution.

Does it have to be this way? Could we have a version of sharing-economy apps that would distribute wealth more equally? Could we have a version of social media that wouldn’t make people so polarized? And is there anything the coders at technology companies can do, or is this dependent on the structure of our economy? These questions are getting a bit beyond the practical and hands-on applications I address in this book, but we should remember that the outcomes of these profound questions are built out of many small design choices.

Data-driven technologies have created new opportunities for harm or help. Many new uses of data-driven and information technologies have made the world a better place, such as moves toward open data in government and assistive automation and diagnostics in medical technologies. However, we are not yet doing a good job of minimizing the many avoidable harms caused by these technologies, such as replications of racially biased reasoning in algorithms trained on data that reflects human bias.

As technologists, data scientists, and computer programmers, we should be thinking about many other examples of unfairness. What do we think about stealing someone’s time and subverting their autonomy? Social media platforms are addictive by design. Such platforms consume hours of the average American’s day, adults and children alike, every single day, and usually without bringing much (or any) joy or knowledge into our lives. Or what do we think of releasing products into the wild that can be repurposed to abuse vulnerable groups? Apps written for smart homes or smart cars have been repurposed into tools of relationship abuse. These are fairness problems too.

These problems of code creating bad behavior aren’t limited to vices in our personal lives and in small private companies. Intensely public moral concerns also exist. Rich nations increasingly fight wars with robotic (and soon algorithmically powered) proxies that seem to endanger mostly those living in poor countries, while those in rich countries tend to reap the economic benefits of these activities as they innovate and sell their inventions for use abroad, often to totalitarian governments that deploy them against their own people. While robotically and algorithmically powered warfare may have some upsides, such as limiting civilian casualties and removing elements of human unpredictability, there are also massive opportunities for abuse.

Clearly, code has to be carefully constructed in such life-or-death situations. We don’t have any assurances from our governments that such code passes muster on normative concerns related to fairness. In fact, journalists even suspect that some countries are exporting these sorts of weapons, not only using them in their own militaries but also selling them to other countries, including aggressive nations known for deploying force.

Likewise, the owners of large data sets and social media empires are finding ways to monetize general observations about humans—observations they don’t have to pay for—and turn them into profitable products or services that may harm the very people about whom the data sets were compiled. Consider two common examples. First, fake news has gone from a tool of government propaganda to an actual business strategy, because the code behind social media financially rewards fake news. Fake news has become particularly pernicious in part because data-driven news and social media algorithms foster content and ideological echo chambers and social bubbles that tend to exacerbate the spread of misinformation. Data helps this happen, and it makes companies rich in the process. This makes it difficult for consumers to really understand what’s going on in the market and what that market is doing to their personal data and their very autonomy.

Second, a common business strategy, resulting from how the internet and all its code are built on top of advertising, is that many applications purporting to do one thing make their money off quite another. Consider the case of an app that advertises itself as a period tracker but then sells information about customers based on its estimation of whether they might be pregnant. These use cases for computer code don’t comport with usual ideas of what’s fair; even if we can very well imagine arguments in favor of the technologies, lawmakers and ordinary people alike seem to agree that something needs to be done. For technical people, the question is, what?

This book will give us tools to talk about these concerns both from a normative perspective (what kind of values do they implicate?) and a technical perspective (what kind of tools can be used to identify and correct problems?). The goal is to write better code in the ethical sense. We want to write code that produces more equitable, secure, transparent, and accurate outcomes.

With just this brief discussion, we can already list general categories for the impacts of technology on fairness. Note that I’ll take a broad view in thinking about fairness right now: equality, safety, and privacy, but also distributional considerations—that is, how do wealth and access get parceled out, and who decides how the allocation happens? Also to demonstrate my argument that the interaction between social and technical factors is an age-old pattern, I highlight both contemporary and past examples of fairness patterns with respect to technology in these proposed categories:

Technical products or design choices that create foreseeable victims in vulnerable populations
- Historical example: factory machinery that required the small hands of children, ultimately creating economic pressures that brought children out of home and school and into the workplace at young ages.
- Contemporary example: smart home devices that can be repurposed for domestic abuse, creating new vulnerabilities. For example, abusers have used smart home devices to lock victims out of their homes or manipulate the lighting. This isn’t the device’s fault, but it introduces a vector for abuse, an extreme form of unfairness.
Technical products that create new victims by devaluing old rights
- Historical example: the rights of way for railroad infrastructure were rapidly devalued with the invention of affordable automobiles and the shift in United States government spending toward increasing the construction of federal roads. This change made cars increasingly useful and valuable over time and railroads and their associated property rights decreasingly valuable. It also shifted wealth geographically as well as between different sets of property holders.
- Contemporary example: ride-share apps taking over markets that previously required expensive licenses to operate, leading to outcomes such as New York taxi medallions losing 80% of their value in four years.
Business structures and organization in technology-driven organizations
- Historical example: the rise of the oil trusts and associated business structures to control the railroads, aiming to reduce or eliminate competition as a result of full control of transportation infrastructure.
- Contemporary example: building and maintaining proprietary data sets as a large company, which may, intentionally or unintentionally, make it far more difficult for new entrants into the market.
Using technology to avoid accountability
- Historical example: building and running high-pollution factories in poor places that don’t have enough resources to enforce environmental laws.
- Contemporary example: storing your data or choosing your customers to avoid the European Union’s General Data Protection Regulation (GDPR).

We’ll focus mostly on algorithmic fairness, with some discussion that more generally applies to digital products. But it’s also worth taking a step back to think about our current enterprise and how it can be situated in the larger history of technology studies.

Let’s start by considering a few important aspects of how we judge fairness and why new code-driven technologies require that we expand our technical toolkit to include fairness-enhancing techniques. We’ll use these considerations to come up with basic rules and goals that will drive the rest of the discussion in this book. Subsequent chapters will aim to operationalize these goals with concrete engineering guidance, in the form of rules of thumb, principles with which to reason, and code examples for achieving specific goals or ensuring the respecting of specific norms.

Community Norms

One of the most obvious ways fairness touches on technology is the way that technology can interrupt, recalibrate, or violate community norms. Sometimes technology makes it a lot easier for someone to be inconsiderate or impose externalities. An externality is an effect imposed on a third party by an action or a decision made by someone who does not experience the effect. A negative externality is a cost imposed on a third party, such as the pollution produced by a car and breathed in not by the driver but by the people in the area through which the car is driven.⁵

Consider a loud piece of gardening equipment, such as a leaf blower. This technology makes it easier for people to clean up their yards, but the ease comes at the expense of imposing quite a bit of noise and air pollution on their neighbors. No one would consider it acceptable for someone to be screaming in their neighborhood, yet most people have no hesitation about using a leaf blower or other loud piece of equipment.

Likewise, a drone may help someone take aerial pictures in a beautiful natural location, but it also creates noise and a visual distraction for people who are seeking out peace and quiet. There’s a good chance the person piloting a drone around a national park would not feel comfortable shouting or playing a radio at the same volume, but this does not factor in when operating the drone.

In these examples we see two ways in which technology can subvert community norms. First, it provides a new channel to transgress old rules, which exposes a lack of social norms or full community consensus regarding the transgression.

Second, a scaling effect occurs in that technology can take one person’s actions and impose their costs on many other people in the community with no additional effort from that person. Consider a recent incident in which a drone shut down Heathrow Airport, a bit of mischief carried out by just one person that inconvenienced thousands and cost airlines and passengers huge sums of money via knock-on effects (secondary or indirect effects). Scaling effects can also apply even without intentional harm, such as when the viral success of Pokemon Go caused significant disruption to and trespassing on private properties where the digital system happened to place game features of importance—despite the lack of any malice on the part of the game’s creators (who likely could not have expected the game’s uptake to be as extraordinarily fast as it was).

These observations aren’t true only for mechanical technologies, such as leaf blowers. They are also true for code. For example, online dating apps offer opportunities for one individual to impose externalities on many others, both one’s potential dates and competitors in the dating pool.

Imagine a coder who boosts their odds of matches by deploying a bot to indicate interest in any and all potential partners, simulating a human rather than investing their time in reviewing possible connections. They enlarge their own dating opportunities, but do so at the expense of everyone. Their potential love interests may devote their own time to connecting with a bot (who may eventually be replaced by the real coder, or not), unbeknownst to them. Also, competitors for the same partners now need to compete with a bot flooding the dating market with low-quality communication. This in turn might lower the quality of the dating app, harming anyone who is giving their time and attention to a dating market where they hope others are playing fair—namely, putting in the same level of effort and attention as they are. In fact, this seems to be one of the gripes of some dating app users: that they cannot get the high-quality interactions they are looking for. This all comes from the coder violating community norms (e.g., be a real person) and possibly also the website’s terms of service.

Technology offers low-cost violations of community norms in many ways. Such opportunities circumvent the usual implied social contract, amounting to low-cost, low-responsibility cheating, given the wrong conditions for technological deployment (such as unduly cheap leaf blowers or poorly designed dating app APIs). Technology isn’t the only reason norm violations occur, of course, but it is an important enabler of such misbehavior. We should recognize all technologies as not merely inventions but also potential incursions into the established social order. Whether this is good or bad should be debated, rather than allowing a social change to be steamrolled into existence through sheer force of code.

Equity and Equality

In the US, we are taught from an early age that “all [people] are created equal,” but what this means to different people can be wildly different. Usually most of us support equity rather than equality, but it depends on the circumstances.

Equity implies that individuals get what they deserve (by some metric of deserving), while equality implies that individuals all get the same thing. While equality is an appealing concept that is quite appropriate in some use cases, it has not proven very practical when used to structure an economy or society. People are unequal in ability even if they remain equal when it comes to deserving the protection of fundamental rights. For this reason, most of the time equality doesn’t survive much intellectual interrogation for practical use cases.

That’s why we write ML algorithms not so we can treat people equally but rather so we can treat them equitably—that is, according to their merit on a metric specific to a given task or purpose. For example, most people like that we earn different incomes for different kinds of jobs. Likewise, we don’t even want all children to be treated equally. If a child has special needs, such as the need for a speech therapist or additional medical treatment, we’d like to give that child extra resources rather than merely resources equal to those of other children.

Equity itself, however, is not a simple one-size-fits-all solution. Equity involves discerning merit—who might deserve more or less depending on the features we think are relevant.

In mainstream US culture, we tend to assume that hard work, talent, and a good attitude are the virtues to be rewarded by meritocracy. But how did we decide these were the values that should be rewarded? Who made the decision, and how did they reach it? These are important questions for at least two reasons. First, these virtues may seem neutral, logical, and intuitive on first inspection, but we have reason to question them. Prioritizing these virtues has tended to favor those with financial resources over those without such resources. It has favored white Americans over others, men over women, etc. So it might be interesting to consider who elevated these qualities, and whether, in doing so, an overly narrow view of the world was employed (one that does not factor in challenges some groups face). Second, we should remember that our current definitions of virtue might ultimately prove outmoded, sociologically naive, or downright illogical when evaluated by other societies. For example, in history we see societies that valued quite different virtues for meritocracies, such as ability to memorize ancient texts, skill in warfare, religiosity, and ability to have children. So we should have some humility, recognizing that our own ideas of meritocracy might not age well. This is not always obvious, and people can disagree even when motivated by the best of intentions.

One powerful example of why society needs to grapple with different notions of merit, and relatedly fairness, comes from the increasing use of algorithms at various decision points in the American criminal justice system—to indicate, for instance, whether an accused criminal can be released on bail or how long of a sentence to impose on a convicted defendant. In 2016 a widely shared news story revealed potential problems with the COMPAS algorithm, which is used to assess recidivism risk when convicted criminals are sentenced for punishment or assessed for early release from prison.

The ProPublica story found that false-positive and false-negative errors for someone labeled high risk to commit a violent crime after release from prison were different for black people and white people. To make this more concrete, imagine a particular black defendant was sentenced with input from the COMPAS algorithm. The ProPublica review found that they would be more likely to be falsely labeled as a high risk for recidivism than if they were white. On the other hand, if a white defendant was sentenced with input from the COMPAS algorithm, ProPublica found that the defendant would be more likely to be falsely labeled a low risk for recidivism. So the algorithm tended to favor white defendants and point in the direction of mistakenly releasing them from prison early—meaning mistakes about white defendants were more likely to result in more leniency, as compared to mistakes about black defendants.⁶

This finding was controversial and garnered a lot of news coverage. Many academics and criminologists admitted the truth of the news story while defending COMPAS as an important instrument for advancing antidiscrimination priorities in criminal justice. These academics pointed out that by other metrics, focusing on individual-level fairness rather than group-level fairness, the COMPAS tool was indeed fair. In particular, the metrics used when originally assessing COMPAS sought to ensure that similar individuals would receive similar treatment regardless of their race.

However, baseline rates of reoffense are different in the white and black defendant populations. Such a difference in baseline rates is itself a result and symptom of racism, but it also means that different models of fairness—either at the group level, as discussed in the ProPublica article, or at the individual level, as measured by the academics—could not both be satisfied. It was not possible to be fair both to individuals and to groups at the same time.

Equity versus equality (in this case, embodied in a debate about individual equity or group parity) has always been contentious, and particularly so with the rise of ML tools applied to a wide range of human experiences and outcomes. The increasing importance of the equity-versus-equality debate results from the increasing potential for transparency and quantization in areas of society traditionally free from such quantitative analysis. As more kinds of important decisions are automated or at least recorded digitally, more decisions can be analyzed, just as happened with the news coverage of the widely used COMPAS algorithm.

Importantly, algorithms in criminal justice are not necessarily bad news; they might lead to greater fairness at the systemic level. While some bemoan the move to algorithmic justice, in fact such a move means that the decisions made can be better monitored for quality and fairness. This is because individual judicial decisions or parole decisions have not historically been recorded in an accessible format subject to systematic inspection and review, whereas automated systems offer the potential for better data compilation practices. This means that we can better describe systems in terms of their performance and then debate what kind of performance we want, hence the equity-versus-equality debate.

Our notions of fairness will be tested and further developed over time. This can be a good thing as technology allows us to better and more precisely articulate how we define fairness and what the standards should be for its implementation. We should hope that the coming decades bring a much better definition of equity and equality to society than we have previously had.

Security

New technologies sometimes make us safer and sometimes make us less safe. Often, whether a technology has a security-enhancing or security-reducing effect is debatable, and can depend on the moment we make the assessment. What’s more, sometimes the net effect is debatable. Like other information technologies, ML applications can result in either more or less safety, depending on their purpose, the quality of their execution, and whether sufficient emphasis was placed on fairness during their development. In some cases the dangers are related to privacy, but in other cases the concerns are related to protecting digital or physical assets from hostile incursions.

Consider physical security. Does a given technology and the accompanying code make us physically safer? That depends. It is predicted that autonomous vehicles (AVs) will be safer than human-driven cars, but that’s little comfort for victims of current deficiencies in ML algorithms, such as an Arizona woman who was killed when an AV’s ML algorithms failed to recognize her as a person or to predict her movement in a situation that seemed a human driver would have handled without trouble.

Property security is another vital element of fairness. A fair world is one in which what we own remains in our control in a way that reflects reasonable ex ante expectations about the nature of a particular kind of property.⁷ In some cases, the property in question is our personal information, in which case it’s a privacy concern. In other cases it’s not about privacy but about keeping what’s ours, be it the financial funds in our bank accounts or the physical electronics we own and the electricity they are consuming. For example, in the case of invasive viruses that co-opt our computers to cryptomine on behalf of someone else, a security failing results in a fairness violation, as victims are paying for resources used by someone else, effectively having their property allocated unfairly and secretly away from their own use.

In both physical and property security, one major concern of technological tools is to prevent their misuse. For example, the COMPAS algorithm discussed earlier was developed to assist with parole decisions (whether to let someone out of prison before they serve their maximum sentence), but has since been used for sentencing decisions (setting the punishment prison term). Some might say that this tool was begging to be used in other applications from the start, as we can imagine administrators of the criminal justice system were keen to find quantitative tools that can reduce the workload of judges, even by deploying a tool in a situation for which it wasn’t designed.

Sometimes tools violate security even when they are used as designed. Consider the case of a “gaydar” tool that was designed to identify sexual orientation from a photograph. While the tool was built as a warning regarding ML algorithms, its very existence may serve as a security threat to anyone whose image might be passed through the algorithm. It could be a security threat both to someone’s physical safety and to their interests in the world, be it their reputation, their employment, or their right to keep private information private.

Privacy

Privacy is deeply linked to security, but it is also a separate value. Here’s a simple example to illustrate. Imagine a company is storing personal data about you. It’s a security problem if that data is found by hackers or is insecurely stored such that it’s publicly available. But it’s a privacy problem if the data is being used in ways you do not expect and have not consented to. In fact, depending on how that data was collected and used, many kinds of privacy infractions could exist.

This should matter to every data scientist, as they should ask themselves whether the analyses they are performing with the data are fair and sensible, given the original purpose for which the data was provided. This should be a particular concern when analyzing metadata, since those providing it are often not even aware that observations of their decisions and actions goes beyond the content they consciously and voluntarily provide.

I do not devote extensive time to taxonomizing privacy, but it’s important to recognize the complexity of the topic and different categories of privacy invasion. Here I follow Daniel Solove’s “A Taxonomy of Privacy” in delineating four broad categories of privacy violations.

Information collection

Information collection is likely one of the main forms of privacy violation that come to mind when this topic comes up. Here are Solove’s general categories of privacy infringements and some ways in which these infringements commonly occur:

Surveillance: Surveillance involves the routine collection of observational data, usually surreptitiously, and characterizes what can fairly be called metadata. In most Western countries, the revelation of surveillance either by the government or by private entities has been met by strong objections from ordinary people and activists alike. However, with the increasing use of digital worlds for much of our waking hours, surveillance has become far less costly and obtrusive. Yet when issues of surveillance emerge in tech, they meet the same widespread criticism and objections. It is far from clear that data subjects are happy about the level of surveillance they experience in ordinary use of digital products, and it’s probably not fair anyway, particularly given the structure of the digital markets, which make it nearly impossible to escape the reach of Big Tech.
Interrogation: Privacy violations can fall into the category of interrogation when they involve directly asking for information. While providing information in response to a query may seem to meet typical expectations of notice and consent, this fails to recognize the realpolitik of many situations in which data is solicited directly. Imagine you are running a company that uses puzzles or surveys on job candidates—do they really have a choice as to whether to answer your questions? Likewise, if you are running an ed-tech company that is adopted by a public school district, do students really have a choice about opting out of any onboarding your product specifies?

Information collection is particularly important when thinking about digital products because so much customization and monetization of digital products is currently driven by data collection that can fairly be characterized as surveillance. This is a pervasive element of designing digital products and building machine learning pipelines, but it’s not clear that these practices are fair or desirable for future technology development.

Information processing

Another category of privacy violations results from the practice of information processing. Separate from the question of how data is gathered is what is done with that data once it is gathered.

The fundamental legal regime worldwide tends to be notice and consent. The idea is that you should use data you collect only in a way that is consistent with the notice you provided and consent you thereafter obtained regarding the data. This is a very lax standard. In practice, it tends to mean that as long as companies honor their own convoluted privacy policies as posted on their websites and the like, they are in the clear legally. So, your legal obligations will often be limited to letting people know what you will do with the data, and you can usually do this with quite broad language.

There are, of course, exceptions to notice and consent. For example, in the US, financial and health data benefits from special protections that set minimum standards for how such data can be collected and what it can be used for. Also, in the EU the GDPR adds enhanced standards to protect data subjects—that is, the people about whom data is collected. Similar measures are in place in other jurisdictions, although they’re notably lacking in the US, home of Big Tech.

The legal protections, however, hardly establish the best definition of fairness. Ordinary people are quite unhappy about the degree of data collected and processed about them, even as they feel they have no choice in the matter. What’s more, even when data is collected under this regime, myriad privacy violations still result from information processing. Some of them are described here, but this is not an exhaustive list:

Aggregation: One problem with information processing is that it generally involves compiling data sets that are not transparent to the data subject even when they have gone through some form of notice and consent. Various data sets may be combined to provide a more complete picture from the data scientist’s perspective or more inputs for a deep learning model, but these create new privacy violations. An individual is then seen in a deeper or more revealing way than was consented to, and the individual’s data is now in a more threatening and powerful format.
Insecurity: A privacy violation results even where a concrete violation has not been identified if data processing results in insecurity of the data itself—for example, lax cybersecurity standards or even the careless handling of data during exploratory analysis by analysts. This kind of insecurity is a privacy violation regardless of whether any information is disseminated, as the data processing has created an enhanced probability of a successful attack, such as identity theft, and therefore heightened risk to the data subjects.
Labeling and discovery: Often the purpose of processing data is to uncover hidden connections and correlations. This is why so many people from different areas of society, from laypeople to technologists, love machine learning and tend to believe it can do far more than it actually can. Many of us are fascinated by the idea of valuable information being out there in the world if only we could unearth it from the data we have recorded. Apart from the problem of identifying spurious correlations in data sets, leading to bad models that can generate bad outcomes, there is also the problem that ML does indeed uncover true facts and correlations that were previously unknown. These facts then tend to undermine previously private information and lessen its privacy. For example, imagine I built an ML model that can use facial recognition to determine personality type.⁸ Now people have lost privacy they used to have, by virtue of such a model existing. Their personality used to be private from mere visual inspection, but now it is not.
Secondary and downstream use: Another privacy violation results when information was originally collected for one purpose but is then deployed for another purpose. Formally, this is likely a violation of notice and consent, but even if the notice given was worded to allow this, it is not clear that the data subject could have consented if the terms are fairly vague. Such secondary use can result in a variety of harms, such as dignitary harms in commoditizing personal data about someone and harms based on violation of the consent originally obtained for such data. It can also mean that data is being used in a way that the data subject might even find offensive, such as if it is processed to make predictions to assist a political candidate a data subject despises.

Information dissemination

Privacy violations also result from various forms, intentional or otherwise, of information dissemination.

Breach of confidentiality: A privacy problem arises if models are being built that unintentionally “leak” your data, as has been shown to be technically possible with a variety of natural language models. Researchers have known about this problem for a long time but have yet to discover a solution. A variety of metrics may be able to indicate the likelihood that this happened—but whenever a model is released, the problem remains that it might be possible to back out actual information about a person whose data was used in the original training set. This area of research is ongoing and one in which industry practitioners need to learn to balance the risk and make reasoned assessments.
Exposure: Exposure results when someone’s information can be identified as the result of the ML work done. For example, it’s a privacy problem if data sets that include your information are released in an insufficiently de-identified form. Given current ML technologies and many opportunities to correlate big data troves, some people question whether data can ever be sufficiently de-identified. This is worrying because as data stores increase, what constitutes sufficient de-identification is a moving target. In many cases—even as personal as someone’s genome—sensitive data has turned out to be insufficiently de-identified.
The human gaze: In some cases, data subjects may be comfortable with data collection and processing if it is done in a purely automated fashion in a large data set, where they imagine enjoying the effective anonymity of not being seen by the human gaze. It’s a privacy problem if humans are observing you or your data when you thought such data would be processed only by computers, as has proven the case with many “home assistant” devices. In such cases, consumers have not realized the extent to which information collected about them was not merely fed into an algorithmic training process but was also inspected by humans who listened to and transcribed various audio recordings, some containing quite sensitive and private information.

Invasion

A privacy invasion occurs not only when you take information from someone or divulge information about someone. It also occurs when you disturb their private moments or private spaces, such as their homes or their very thoughts, attention, and experiences.

Intrusion: It is a privacy violation when your product intrudes into what should be a private space in a way that lessens its intimacy, safety, or security, be it a physical or figurative space. For example, a robotic vacuum cleaner that turns itself on and wanders into a bedroom uninvited is a privacy intrusion. But so is a pop-up notification on someone’s phone that does not have permission or justification for the interruption.
Decisional interference: When data is used against someone’s interest, such as to convince someone to buy more food than they want or to watch more television than they think wise, this may not be a good modeling. This could be a privacy invasion, as well as a separate harm of invading privacy to work against someone’s interests. Incidentally, a legal idea is making the rounds that information fiduciaries should not be allowed to work against the interests of those about whom they have information.

Data and Fiduciary Relationships

A fiduciary is a person who is in a position of trust with respect to another person and is therefore expected to put that other person’s interests ahead of their own. So, for example, fiduciary obligations are imposed on lawyers and doctors when advising their clients and patients. For large tech companies with troves of personal data, the idea of an information fiduciary has arisen: since these companies have so much data and so much intimate access to and potential knowledge about their data subjects fiduciary duties of loyalty should apply. Note that this is not the current state of the law but rather a proposal by Balkin (2016)⁹ as one means to recognize and respond to the growing discontent with the power and reach of many companies that have made their mark on the market through personal data.

A way of thinking about privacy: contextual integrity

One theme we can see in all of these categories of privacy violation is that they tend to upend norms and normal conceptual understandings of how to classify behavior and whether that behavior is acceptable. One theory that has been advanced to unify thinking about privacy norms and privacy violations is contextual integrity, a theory developed by Helen Nissenbaum in her seminal work, Privacy In Context (Stanford Law Books, 2010).

Contextual integrity is a useful benchmark as well because it can offer practical guidance appropriate to practicing technologists. Contextual integrity comprises four essential claims:

A privacy-protecting environment permits information flows in appropriate channels.
Appropriate channels and directions of information flow conform to informational norms, which are highly context-specific.
Context-specific norms are assessed by looking at five situational parameters:
- Identity of the data subject, the person about whom data is collected
- Identity of the sender of that information
- Identity of the recipient of that information
- Content of that information
- The transmission principle, which reflects the rules of operation for a specific chain of information transition, such as the directionality of information flow and the expected ability to propagate the information forward, or not, as specified by, for example, expectations of confidentiality
Evolution over time as to ethical concerns and contextual norms for information sharing as culture changes or expected practices evolve, as can happen due to technical or nontechnical factors.

Contextual integrity was fashioned to provide a practical approach to considerations of data collection and analysis policies. Contextual integrity thus takes a pragmatic view, recognizing that fully explicit notice and consent is not only unworkable, given the volume of data stored and analyzed in our digital lives, but also undesirable, given that people will not always find it normal to ask them about basic data collection practices that are widely accepted and judged reasonable or even desirable. So contextual integrity offers a path for privacy analysis that can reduce the friction of notice and consent but also enhance privacy and feelings of appropriateness relative to the notice-and-consent scheme currently governing most electronic data collection policies around the world.

I discuss specific mathematical privacy metrics in Chapter 2. However, the conceptual tools offered by a taxonomy of privacy violations, and one proposed method (among several others) for making assessments as to assessing whether appropriate uses of data, are also important for understanding this very broad and important topic .

Legal Responses to Fairness in Technology

Concerns about technology and fairness go back a long way, even from a legal perspective. For example, as early as the 1970s it was illegal under French law to make any decisions affecting human beings in a purely algorithmic manner—that is, without any human supervision. Such concerns about machines regulating humans without any human oversight are also reflected in more recent laws, such as the GDPR, which likewise introduces protections against automated decisions.

Similar legislative proposals are also appearing in the US. For example, Washington State has drafted legislation that would issue strict guidelines regarding the appropriate use of, and in some cases prohibit, state governmental agencies from using automated decision-making algorithms in important decision contexts.¹⁰ At the federal level, recent legislation has been proposed to address both potential bias and a lack of accountability and transparency in ML systems. One example of such legislation is the Algorithmic Accountability Act of 2019; see Figure 1-1.¹¹ (These are examples of proposed legislation. However, such legislation has, so far, mostly not made it past the stage of a proposal in the US federal and state governments, with some notable exceptions that I will discuss in Chapter 12.)

Unlike the EU, the US does not have any national law in force with respect to algorithmic decision making or data privacy, both of which are covered by the GDPR. In contrast, many Asian jurisdictions are more similar to Europe, such as China and Singapore, which are both actively developing targeted national legislation and regulations on privacy and ethical uses of ML (and artificial intelligence more generally).

On the other hand, traditional security concerns are more systematically and widely addressed in the US, perhaps because they constitute a longer-standing formalized technological concern than do privacy, discrimination, and fairness. The US has national information security laws going back at least to the 1990s, and the US military plays an active role in setting cybersecurity standards that have an international impact, such as for the Internet of Things (IoT).¹² Similarly, governments in other high-tech jurisdictions are keen to assist with setting security standards in technological industries.

As technological fields mature, we see more concrete laws being made about these technologies. We also see technical consensus emerging about best practices and definitions of standards. However, security protocols will continue to evolve because new technologies create new security concerns.

Over time, we can hope to see more consensus from a technical perspective as to what constitutes “fair” technology, particularly with respect to data analysis, machine learning, and more generally the field of data-driven AI. We can expect that as technical standards emerge from community agreement, these standards will allow more consistent and reasonable social and legal expectations. For those in technical positions, you will find yourself having power over the products you design but not necessarily a perfect way to ensure fairness. Indeed, fairness questions remain an area of active discussion and debate within both the legal and the technical communities that think about these problems. However, even if a consensus about best practices has not yet been reached—and may never be—this book will take you through a variety of options, all of which are better than continuing to ignore the problem.

We will discuss rules and standards more toward the end of the book. Many ML-related efforts are underway in this domain, and it’s a rapidly changing field, so my goal will only be to make you aware of some interesting examples. It will not be possible to offer a full rundown of what might apply to your own ML practices depending on where you work and where your data subjects and product users are located.

The Assumptions and Approaches in This Book

This book will no doubt meet with many criticisms of inaccuracy, oversimplification, understatement, overstatement, and so on. I am attempting perhaps a fool’s errand in reducing “fairness” to a coding book of only a few hundred pages. So I’d like you to have the same understanding regarding the goals of this book and my perspective in writing it.

My perspective on fairness is as a US-trained lawyer who has also worked as a coder and data scientist. Most, but not all, of the anecdotal examples and historical motivations I use to guide this book are grounded in US history. That is not because I believe the US is the center of the world, but because this background is the foundation and because the US is one large and important center for the tech industry.

The US is also reasonably representative in the nature of problems that arise everywhere, even if the situational details have different names or different histories. The problems we have here in the US are the problems societies all around the world have in one form or another. The US has problems with gender discrimination, racial discrimination, and discrimination on the basis of other important categories such as religion, disability status, and sexual orientation. We in the US, as in other parts of the world, also recognize other elements of fairness, such as a social environment that respects individual rights to privacy and individual and community needs for security. The concepts explored in this book are of universal interest; all societies ask the same fundamental questions about fairness, even if the dominant or preferred answer in a specific situation varies with culture or geography.

The particular groups affected by fairness concerns in different societies will be different, no doubt, but the senselessness, cruelty, and unfairness of, for example, discrimination will surely be a unifying theme. So wherever you are reading this, please be understanding when I refer to certain historical trials in the US. I would invite you to either substitute in your knowledge of your local context or, if you don’t have that knowledge, educate yourself (and me) on your local context. I welcome feedback from readers and look forward to making revisions and additions to this text to provide a more global perspective on fairness (see the Preface for contact information).

Relatedly, I assume my readers have many commonalities with me, even if we don’t come from the same country. I assume that all groups of people should be treated equally and that we should come into any problem affecting humans with the assumption that ability is spread evenly throughout the human population.¹³

When I have given talks or taught on topics related to fairness, a question inevitably arises: “What if, for some reason, ability for a given trait is not spread evenly throughout the population?” After all, certain regions of the world seem to dominate certain activities, be it chess playing or marathon running. Perhaps, some like to suggest, there is some genetic component. Shouldn’t we account for these sorts of inherent ability issues somewhere in a topic of fairness—that people who actually are better should be recognized as such?

I’d like to point out that for my purposes this is neither an interesting nor an important question. I’ll quote Neil deGrasse Tyson’s response to a question on a panel about whether women might be less present in the sciences because of ability or genetics. His response got the to meat of the issue:

I’ve never been female, but I have been black my whole life. So let me perhaps offer some insight from that perspective, because there are many similar social issues related to access to equal opportunity that we find in the black community as well as in the community of women in a white-male-dominated society…

[T]hroughout my life, I’ve known that I wanted to do astrophysics, since I was nine years old on a first visit to the Hayden Planetarium…I got to see how the world around me reacted to my expression of these ambitions. And all I can say is, the fact that I wanted to be a scientist, an astrophysicist, was, hands down, the path of most resistance through…the forces of society. Anytime I expressed this interest, teachers would say, “Don’t you want to be an athlete?” I wanted to become something that was outside of the paradigms of expectation of the people in power…

So, my life experience tells me that when you don’t find blacks in the sciences and you don’t find women in the sciences, I know that these forces are real and I had to survive them in order to get where I am today. So before we start talking about genetic differences, you’ve got to come up with a system where there’s equal opportunity. Then we can have that conversation. (Emphasis mine.)

If your interest in this book lies in disentangling questions such as, “To what extent are men better than women at science and to what extent is it unfairness that stops women from being better represented in the ranks of Nobel-Prize-winning scientists?” this isn’t the book for you.¹⁴ On the other hand, if your question is, “To what extent can I design and build digital products and data analysis processes that ensure people get equal opportunities, thus contributing to a world that looks more equal to everyone?” then this book might be a good starting point.

What If I’m Skeptical of All This Fairness Talk?

My audience always includes fairness skeptics when I give industry talks about fairness practices for ML and digital products. The questions they raise tend to be of two flavors. One flavor is, “But what about treating people according to their merits?” The other is, “But isn’t fairness a sideshow in innovation?”

With respect to the first question, yes, fairness is all about treating people according to their merits (alongside other core principles discussed in Chapters 1 and 2). I addressed this concern in “Equity and Equality”. So now I turn to the second question.

Won’t Fairness Slow Down Innovation?

Short answer: no. In fact, fairness itself is in the midst of a revolution in law, computer science, mathematics, and behavioral sciences. Concerns about fairness are generating new mathematics, new programming, and new legal discussions. If anything, not thinking about fairness for much of the beginning of our current information revolution arguably held up an area ripe for the innovation we are seeing now.

My other rebuttal to this concern is the following:

The best minds of my generation are thinking about how to make people click ads. That sucks.

Jeff Hammerbacher

While I don’t fully subscribe to the idea that all the “best minds” have gone to the dark side, I share more than a little of Hammerbacher’s concerns that the outsize profits that have been generated by uninspiring technical or behavioral science progress in ad clicks may hardly be the kind of “innovation” our society should be allocating resources to. I am not trying to make a political statement about how we should allocate resources, but surely we can agree that much of what passes for “innovation” in digital products is really just aggressive direct-to-consumer marketing. So let’s keep our high horse about innovation in check—if we’re going to say that basic fairness measures put a damper on tech profits, it’s not necessarily a damper on socially beneficial innovation.

Are There Any Real-World Consequences for Not Developing Fairness-Aware Practices?

Short answer: yes. In Chapter 12, I discuss some of the laws that regulate data-driven digital products. While I don’t like to emphasize the negative aspects of such rules, you should be aware that violating these laws carries real-world monetary consequences. Most notably, the GDPR’s penalties can rise to as high as 4% of the annual global revenues of a firm for severe infringements, and as high as 2% of annual global revenues for less severe infringements. The penalties can scale with the revenues of a firm precisely to make sure that the penalty will be felt regardless of the size of the organization.

In 2018, over €400,000,000 of fines were assessed, and in 2019 over €440,000,000 were assessed. One notable fine was €50,000,000 against Google for having opaque and inaccessible consent policies, which pertains to the fairness principle of transparency and consent. Another notable fine was €18,500,000 against the Austrian national postal service for processing information related to political affiliation, package receipt frequency, and frequency of relocation for the purpose of direct marketing, a violation of the lawful basis of data processing under the GDPR. We can expect to see many more fines in the future due to intentional and unintentional violations of this wide-ranging law, which establishes some rules governing the fair use of personal data and algorithmic decision making.

In the US, lawmakers have been notably inactive in this area, but companies can still face legal penalties for recklessly disregarding basic fairness ethics and best practices. For example, in 2019 the Federal Trade Commission fined Facebook $5,000,000,000 (that’s $5 billion) for a variety of improper data practices and misleading statements in its privacy policy that were found to constitute unfair and deceptive practices. These were related to how the code that powered the social media website differed from representations in its privacy policy.

Separately, the US Department of Housing and Urban Development sued Facebook in 2019 for housing discrimination in violation of the Fair Housing Act. Specifically, Facebook was allowing advertisers, even for housing, to target audiences according to a variety of factors that could relate to protected categories, such as race, religion, gender, and family status. In this case, Facebook escaped financial penalties but reached an extensive settlement agreement that detailed the ways in which it had to immediately reform its advertising services to conform to federal law and basic fairness principles.¹⁵

In yet another recent defeat for Facebook, in 2020 it elected to settle a class-action lawsuit, Patel v. Facebook, that came about as a result of violations of the Illinois Biometric Information Privacy Act,¹⁶ a law had been passed years earlier but that had remained underused until recently. The law gives a private right of action for nonconsensual storage of biometric information, which may include information related to facial recognition, a very commonly applied use case in social media as well as in other domains. Facebook will pay out $550,000,000 (that is, more than half a billion dollars) to settle this claim.

I don’t cite all these actions against Facebook to scapegoat one company but to point out that years of violating laws can come home to roost, as appears to be the case for Facebook. Realistically, it’s possible that these amounts haven’t been especially damaging to Facebook (for example, it readily paid the $5 billion FTC fine in cash, without raising the funds with loans and such), but such large amounts might at the least be noticeable even to a hugely profitable company such as Facebook, and they certainly would be for smaller companies.

What’s more, in both the EU and the US we are seeing a “techlash”: a populist rising up against these companies that demonstrates pent-up resentment with both their policies and their financial success, which seems to come about in part from violating consumer consent and fairness principles. Many antitrust (in EU, competition law) investigations are ongoing against Big Tech companies, partly motivated by a sense that these companies may be violating basic notions of fair play against both consumers and competitors.

So the short answer is yes, real-world penalties exist even if you don’t find fairness a particularly compelling ethical consideration when you build digital products. And given the political climate of late 2020, we can expect to see ramped-up penalties in future years. So even if you play by the rules of realpolitik, it’s a good time to get some basic fairness hygiene into your digital products.

What Is Fairness?

Fairness can mean many different things to many different people, as to both what concerns are fairness concerns and which way the balance of fairness tilts. I don’t seek to conclusively and extensively define fairness, nor do I seek to debate each nuance. However, fairness has many operational concerns, particularly with respect to fairness in ML systems, which are important to recognize and are discussed in various ways throughout the book. I describe these next and highlight which areas of the book most address this issue.

In the next section, I narrow down the elements that will be most extensively discussed in this book to the following:

Antidiscrimination (which I also treat as the most important concern of equality or equity)
Security
Privacy

I focus on these three topics for a few reasons. First, these topics have extensive backgrounds of both legal and technical guidance, suggesting that they are particularly ripe for discussion and for the emergence of basic knowledge and good practices expected of every software engineer, data scientist, and ML engineer. In these mature fields, unfortunately, it has been acceptable for far too long to ignore these three issues despite their well-established legal and technical basis for identifying and resolving fundamental concerns.

Second, these concerns cover what appears to affect ordinary people most right now, particularly if we use newspaper coverage as guidance. This is not to say that these are the only concerns, but for the moment they remain the main talking points in the popular media.

Also, as I detail next, the book covers other topics, so I make clear here which chapters address topics in various domains of fairness:

Antidiscrimination: Antidiscrimination seems to receive the most attention in the press and in political discourse when it comes to fairness in ML systems. Antidiscrimination is also extremely concerning with respect to any system because unlike some fairness concerns, which would implicate everyone subjected to a system, discriminatory outcomes often affect only a small group, and usually a disfavored group. Similar to both press coverage and popular discourse, this book therefore gives quite a bit of attention to antidiscrimination. Particular attention is found in Chapters 2 through 6.

Privacy

Privacy has long received attention both in the law and in the technology sector as a salient fairness feature for many consumers. It also seems to form a large component of the techlash, which continues at the time of writing and shows that consumers are particularly resentful of incursions into their private spaces by technology and ML modeling. Particular attention to privacy is found in Chapters 2 and 9.

Safety

Safety is often taken for granted in digital systems but is, in fact, woefully lacking in most of them, particularly those powered by ML models. Cybersecurity will continue to be a key issue because of the very nature of the beast, one in which continued evolution and an arms race will be a routine part of digital security. However, the importance of security related to ML modeling is underappreciated, even by those who develop and release ML models for industry. Such lapses have particular importance now as ML models increasingly have physical real-world implications.

While security and safety are generally treated as distinct from other fairness attributes, this book makes the case that safety is part of fairness and very much part of what an individual should be able to reasonably expect as we think about introducing digital products into the human environment. It would be terribly unfair to make the world less fair all in the name of digital progress. Particular attention to safety is found in Chapters 2, 10, and 11.

Transparency

Transparency is a key value in the rule of law but not necessarily an acceptable practice in all areas of business. For this reason, as ML models and algorithmic decision making become more widespread, we see that calls for transparency have made their way even into areas traditionally occupied only by the private sector. As digital venues and products dominate important markets, sometimes providing the only venues in which speakers or writers can hope to reach sizable audiences, and as businesses enjoy an outsize role in cultural and political life, we can expect to see more calls for transparency regarding many elements of digital product design and the inspection of ML pipelines. Particular attention to transparency is found in Chapters 7, 8, 11, and 12.

Legitimacy

Legitimacy, derived through democratic process, conformance to fairness norms, or other procedural- or outcome-based metrics, is important to the justification for any rollout of a digital system, and particularly one that could have high-stakes ramifications. The techlash of recent years suggests that in many cases the business models and the data practices surrounding data-driven models have suffered from a lack of legitimacy, as such products are perceived as rolled out to the benefit of elites and to the detriment of and against the explicit objections of vulnerable populations. Particular attention to legitimacy is found in this chapter and Chapters 11 and 12.

Accountability

Accountability relates to the need for feedback and consequences from downstream uses of digital products and ML models to make their way back upstream to their developers and designers. In the current digital marketplace it is too often the case that those responsible for digital harms suffer few consequences as a result of their actions. This in turn can mean, first, that too little care is taken to anticipate and avoid harms and that, second, too little is done to remediate harms even when they become known. Particular attention to accountability is found in Chapters 7, 10, 11, and 12.

Autonomy

Digital products can appear to or actually take away autonomy from humans. This can happen in a number of ways. In some cases, computers stand in where humans used to make decisions, removing human cognition and discretion from a system. In other cases, digital products are used to cabin the options open to humans either as decision makers or as data subjects facing an ML model. Finally, digital environments themselves add new possibilities to the lived human experience while simultaneously removing others. These practices can all potentially undermine aggregate human autonomy and individual human autonomy if done carelessly or unfairly. Particular attention to autonomy is found in Chapters 2 and 11, with respect to the design of digital systems.

Minority rights

By simple definition, ML can look like the tyranny of a data set’s majority over its minority. More explicitly, minority rights are discussed here with respect to antidiscrimination in Chapters 4 through 6, and then more generally as a concept in Chapter 11.

Distributional concerns

One area of concern discussed earlier in this chapter is that technology can unbalance the distribution of wealth among elements of society. This must be an area of fairness concern, but it is one that can at times be difficult to account for by an individual entity or ML engineer. Nonetheless, these distributional concerns can factor into product design, and for that reason particular attention to them is found in Chapters 11 and 12. Distributional concerns are particularly difficult to study, and we can expect to see more developments on this topic in years to come (and in any subsequent editions of this book!).

Rules to Code By

Here I formulate rules that are generally applicable and widely supported, and these will motivate the aims I pursue in the book’s conceptual and coding discussions.

Equality and Equity

People should be treated equally when doing so makes sense.
- Equality should be defined in a way that is reasonable and meaningful, given reality.
- Equality is not always equitable. Providing equal resources when need is unequal will often be unfair rather than fair.
People should be treated equitably whenever equal treatment would be unfair or unjustified.
- Only meaningful distinctions should be made between people, and these distinctions should be clearly documented and limited in scope.
- When we determine a merit metric for the purpose of treating people equitably, we should clearly define the metric and make sure we are accurately measuring that metric.
- All relevant and reasonably available factors should be considered when sorting people.
- When we write code separating people into categories, it should serve their interests and the interests of the whole community.
Equity and equality should be evaluated over time, with room to evolve.
- Systems should not be deterministic, and should recognize that individuals can change, and make room for that.
- Systems should recognize that correlations and causes can evolve over time.
- Systems should be quality checked over time and should not create self-perpetuating models of reality.

Security

If our code can affect the physical world, it should make people more safe rather than less so.
- Are we sure of our ability to fully test our software before we allow it into the wild?
- Are we sure our software outperforms the older or nondigital system we are replacing, and can we prove it?
If our code affects digital assets, it should match the reasonable expectations of the users. Builders of such systems have an affirmative duty to identify these reasonable expectations and act on them.
- Do our product interfaces fully convey their function?
- Do our products explicitly or implicitly overpromise the benefit to the user?
We should not build tools that can be easily misused to threaten either the security of individuals or their rightfully owned assets.
- Are we building a product that has potentially abusive use cases? If so, what if anything can we do to limit these potential use cases? Should we build this product before we know how to control it?
Threats to autonomy, freedom of movement, freedom of expression, and other values supported by long-standing cultural norms are threats to security.
- Are we building products that an abusive entity would love to have? If so, do the potential benefits outweigh the costs of bringing this technology into the world? Are we simply building this product because we assume our competitors will do the same?

Privacy

It is reasonable for people using systems to determine what is private, not for us to determine what we can find out.
- If we don’t know our users’ expectations, can we find out? How are we educating them about our understanding of their expectations?
The potentially superhuman capacities of ML products should not be used to technologically overrule social norms about what is private.
- Are we respecting known social norms and taking actions to discover latent social norms?
- Are we rolling out a product across too many national or cultural boundaries without adapting it locally? Can we justify a uniform design?
If a product undermines autonomy, sense of identity, or sense of confidence in oneself or one’s environment, it is privacy invading.
Invasion of privacy is a harm even if the person about whom information is deduced is not aware of the observation or could not deduce what is deduced by a technological system.
- Are we taking on a role that would mostly be appropriate for a trained professional covered by codes of professional ethics (especially physician or mental health provider)? Or the role of an intimate friend?
- Would our product be acceptable if replaced by a human doing exactly the same thing? If not, is this the explicit value of the program or an unintended and manipulative side effect?

¹ This phrasing is from Shoshana Zuboff’s work The Age of Surveillance Capitalism (PublicAffairs, 2019).

² Of course, this is quite a simplified presentation of a very complicated history.

³ Khosrowshahi, Dara. “I Am the C.E.O. of Uber. Gig Workers Deserve Better.” The New York Times, August 10, 2020. https://www.nytimes.com/2020/08/10/opinion/uber-ceo-dara-khosrowshahi-gig-workers-deserve-better.html.

⁴ Smith, Brad. “Facial recognition: It’s time for action.” Microsoft.com, December 6, 2018. https://blogs.microsoft.com/on-the-issues/2018/12/06/facial-recognition-its-time-for-action.

⁵ Externalities can also be positive, such as when a person practices her piano playing and a neighbor benefits by enjoying the practice session.

⁶ Of course “mistake” itself is a problematic word if we recognize the inherently probabilistic and uncertain nature of these predictions. After all, for most convicted criminals, whether they will reoffend reflects quite a complex mixture of individual characteristics and environmental influences, so that it’s unclear that a label that turned out to be wrong means there was a “mistake” when it really could just reflect the probabilistic nature of the guess. Still, it remains problematic that these guesses point toward different outcomes depending on a defendant’s race.

⁷ Ex ante expectations refer to what we can expect before an event, using background knowledge and probability. In contrast, ex post knowledge and expectations reflect our judgments given information about what did in fact happen. Hindsight bias is the human tendency to judge ex ante expectations harshly and not fairly, relating to a point of view that was reasonable before events rolled out in a certain way.

⁸ I do not mean to suggest such a model could be built, and believe those interested in fairness should be especially skeptical of the new digital physiognomy.

⁹ J.M. Balkin, “Information Fiduciaries and the First Amendment,” UC Davis Law Review, 2016, https://lawreview.law.ucdavis.edu/issues/49/4/Lecture/49-4_Balkin.pdf.

¹⁰ The Washington State legislation was referred to a subcommittee and never reached the floor of the main legislative body for debate in the state senate.

¹¹ The Algorithmic Accountability Act of 2019 was referred to the Senate’s Subcommittee on Consumer Protection and Commerce, and no further action has been taken. This is often the case with legislative proposals generally, and has so far proven to be the case for all algorithm-specific bills proposed at the federal level.

¹² The Internet of Things is a term used to describe the many devices deployed in our physical world, in either private or public spaces, that connect to the internet and provide information based on physical measurements, be these auditory recordings, image feeds, or other sources of information. These devices can also interact with humans or provide outputs for information or actions having effects at their location.

¹³ This does not mean that it is distributed uniformly and identically in each person, but rather that you will find talented and hard-working individuals in any part of the world and occupying any body type and shape, and not-so-talented, not-so-hard-working people in any part of the world, occupying all manners of body types and shapes.

¹⁴ Note, plenty of work attempts to answer this question, despite complaints that some topics are forbidden because of political correctness. I simply don’t think these are important questions for our time, or possibly ever. Ultimately, don’t we all want a world where each person is judged on their own merits? If we can achieve this, what would be the use of such questions?

¹⁵ Given that such targeted advertising practices were available for years before the lawsuit, some might say Facebook already got away with quite a lot of unfairness and reaped the financial rewards. Unfortunately, the legal system does not always move as quickly as would be desirable to ensure basic fairness. Also, to be clear, in many of these cases Facebook (or other entities) does not necessarily admit wrongdoing, so some of what I write should perhaps be qualified in terms of allegations rather than facts, but I try to keep the language simple here.

¹⁶ Note that this is a state law and does not apply across the US to all US citizens.

Get Practical Fairness now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Practical Fairness by Aileen Nielsen

Chapter 1. Fairness, Technology, and the Real World

Fairness in Engineering Is an Old Problem

Our Fairness Problems Now

Community Norms

Equity and Equality

Security

Privacy

Information collection

Information processing

Information dissemination

Invasion

Data and Fiduciary Relationships

A way of thinking about privacy: contextual integrity

Legal Responses to Fairness in Technology

Figure 1-1. In recent years lawmakers around the world have begun drafting legislation to address a host of concerns related to the rise of algorithmic decision making as an increasingly common technology in business and government alike

The Assumptions and Approaches in This Book

What If I’m Skeptical of All This Fairness Talk?

Won’t Fairness Slow Down Innovation?

Are There Any Real-World Consequences for Not Developing Fairness-Aware Practices?

What Is Fairness?

Rules to Code By

Equality and Equity

Security

Privacy

Don’t leave empty-handed

It’s yours, free.

Check it out now on O’Reilly