Reliable Machine Learning

Chapter 9. Monitoring and Observability for Models

By Niall Murphy and Aparna Dhinakaran

Contributors/Reviewers: Ely Spears, Lina Weichbrodt, Tammy Le

Diagrams: Joel Bowman

Managing production systems is somewhere between an art and a science. Add the complexities of ML to this hybrid discipline, and it looks less like a science and more like an art. What we do today is very much a frontier, rather than a well-defined space. Despite that, this chapter outlines what we know about how to monitor, observe, and alert for ML production systems, and makes suggestions for developing the practice within your own organization.

What Is Production Monitoring and Why Do It?

This chapter is about how to monitor systems that are doing ML, rather than using ML to monitor systems. The latter is sometimes called AIOps; we are focusing on the former.

With that out of the way, let’s talk about production monitoring generically, without the complexities of ML, so we can make things easier to understand—and where better to begin than with a definition? Monitoring, at the most basic level, provides data about how your systems are performing; that data is made storable, accessible, and displayable in some reasonable way. Observability is an attribute of software, meaning that when correctly written, the emitted monitoring data—usually extended or expanded in some way, with labeling or tagging—can be used to correctly infer the behavior of the system.¹

Why would you care? It turns out there are lots ...