97 Things Every Application Security Professional Should Know

Chapter 86. Mitigating Bias and Unfairness in AI-Based Applications

Angelica Lo Duca

AI continues to revolutionize the world, and there is a growing concern for bias and unfairness in AI-based applications. In practice, bias in AI refers to the presence of systemic and unjustified preferences or prejudices in AI systems. Bias can be introduced to data sets in the traditional way, by being present in the raw data (because of environmental circumstances, accidental omission/addition, etc.). But it can also be introduced by malicious actors who want to skew the results of an AI-based app. In contrast, unfairness refers to the actual outcomes resulting from such biases, leading to unequal and discriminatory treatment of individuals or groups. Bias is the underlying issue, while unfairness is the consequence of that bias. As such, it’s important to protect your AI-based apps against that added bias and unfairness in AI training models.

Among the most popular threats in this field, there are poisoning attacks, which are a significant concern for AppSec because they could compromise the AI model. If an adversary understands the weakness of the biased model, they can produce carefully crafted inputs to cause the model to make incorrect decisions. You can also have malicious actors who inject unfair inputs into training data to manipulate AI models to produce targeted responses, such ...