Model Performance Management with Explainable AI

Chapter 2. Explainable AI

Explainable AI (XAI) is a form of AI that aims at creating machine learning models that are, for the most part, explainable and/or interpretable by humans. XAI evolved out of the need to break open the black box of AI models to make them interpretable by humans, with the intent of minimizing the risk of unknown or unpredictable outcomes from those models. XAI is not only relevant for regulatory and legal reasons, but it is also an important tool for monitoring and managing model performance.

In this chapter, we will discuss who in your company (or outside it) might want an explanation of your models’ predictions, the many reasons you might want your models to be explainable and/or interpretable, and how different types of models can be explained. The focus will be on how explainability can be used to understand and thereby improve the performance of ML models.

Explainability in Context

Before exploring XAI, let’s briefly discuss what AI is and its relationship to Responsible AI. AI is a form of intelligence demonstrated by machines, which is akin to natural intelligence demonstrated by animals and humans but without the ability to display emotions or consciousness. You might have heard of some advanced AI systems, such as the autopilot feature on planes or the autonomous driving capability of cars, that have been in the limelight of the AI community for the past decade.

AI systems can be as simple as the devices that turn on the lights as you walk into ...