Machine Learning Production Systems

Chapter 9. Interpretability

Model interpretability helps you develop a deeper understanding of the workings of your models.

Interpretability itself does not have a mathematical definition. Biran and Cotton provided a good definition of interpretability. They wrote that systems, or in this case models, “are interpretable if their operations can be understood by a human, either through introspection or through a produced explanation.” In other words, if there is some way for a human to figure out why a model produced a certain result, the model is interpretable.

The term explainability is also often used, but the distinction between interpretability and explainability is not well-defined. In this chapter, we will primarily refer to both as interpretability.

Interpretability is becoming both increasingly important and increasingly difficult as models become more and more complex. But the good news is that the techniques for achieving interpretability are improving as well.

Explainable AI

Interpretability is part of a larger field known as Responsible AI. The development of AI, and the successful application of AI to more and more problems, has resulted in rapid growth in the ability to perform tasks that were previously not possible. This has created many great new opportunities. But there are questions about how much trust we should place in the results of these models. Sometimes there also are questions about how responsibly models handle a number of factors that influence people ...