Explainable AI for Practitioners

Foreword

When developing machine learning (ML) models, I am sure all of you have asked the questions: Oh, how did it get that right? or That’s weird, why would it predict that? As software engineers, our first instinct is to trace through the code to find the answers. Unfortunately, this does not get us very far with ML models because their “code” is automatically generated, not human-readable, and may span a vast number (sometimes billions!) of parameters. One needs a special set of tools to understand ML models. Explainable AI (XAI) is a field of machine learning focused on developing and analyzing such tools.

Model explanations are not just a nice-to-have feature to satisfy our curiosities about how a model works. For practitioners, it is a must-have to ensure that they are not flying blind. Machine learning models are notorious for being right for the wrong reason. A classic example of this, discussed in this book, is that of a medical imaging model where explanations revealed that the model relied on “pen marks” on X-ray images to make disease predictions.

The rise of ML models in high-stakes decision-making has sparked a surge in the field of XAI with a plethora of techniques proposed across a variety of data modalities. The vast number of available techniques has been both a blessing and a curse for practitioners. At the heart of this issue is that there is no such thing as a perfect explanation. A good explanation must balance faithfulness to the model with human intelligibility ...