Machine Learning Production Systems

Chapter 16. Model Monitoring and Logging

By now, you should be familiar with the MLOps modeling lifecycle, as shown in Figure 16-1, which starts with building your models but doesn’t end with deployment.

Figure 16-1. The MLOps lifecycle

The last task, monitoring your model in production, is an ongoing task for as long as your model is in production. The data you gather by monitoring will guide how you build the next version of your model and make you aware of changes in your data and changes in your model performance. So, as you can see in Figure 16-1, this is a cyclical, iterative process that requires the last step, monitoring, in order to be complete.

You should note here that this diagram is only looking at monitoring that is directly related to your model performance, and you will also need to include monitoring of the systems and infrastructure that are included in your entire product or service, such as databases and web servers. That kind of monitoring is only concerned with the basic operation of your product or service, and not the model itself, but it’s critical to your users’ experience. Basically, if the system is down, it really doesn’t matter how good your model is.

The Importance of Monitoring

An ounce of prevention is worth a pound of cure.

Benjamin Franklin

In 1733, Benjamin Franklin visited Boston and was impressed with the fire prevention measures the city ...