Operating AI

CHAPTER 5Operating AI Is Different from Operating Software

AI model monitoring is an operational stage in the machine learning (ML) life cycle that comes after model deployment. It refers to monitoring your ML models for things like errors, crashes, and latency, but most importantly, to ensure that your model is maintaining a predetermined desired level of performance in production. Monitoring ML models is an essential feedback loop of any MLOps system to keep deployed models current and predictions accurate, and ultimately to ensure that the ML models deliver value long term. Therefore, it's vital that the setup of your model monitoring has a good balance between these different aspects.

Observing and monitoring AI models in production, however, is often an overlooked part of the ML life cycle, almost like an afterthought, whereas it should be seen as critical to a model's viability in the post-deployment phase. Models have an afterlife of viability, and that viable life in production needs a constant watchful eye. Because AI is built on continuous learning principles, it requires another operational support model than traditional software. The feedback loop becomes fundamental, along with highly automated monitoring of model performance and data quality.

Model Monitoring

Model monitoring is vital to detect and address up front, since your models will degrade over time as you use them, a process known as model drift. Model drift, also known as model decay, refers to the ...