Enterprise AI in the Cloud

19Monitoring Models

What gets measured gets managed.

—Peter Drucker

This chapter deals with the critical components of monitoring, including making informed decisions around real-time and batch monitoring. It delves into checking the health of model endpoints, selecting appropriate performance metrics, and ensuring model freshness. To ensure the AI systems remain agile and responsive, you learn how to review and update features, automate endpoint changes, and scale on demand (see Figure 19.1).

FIGURE 19.1: Monitoring models

MONITORING MODELS

Monitoring models is an essential aspect of MLOps to ensure that the model performs well and that the data used is high quality.

Importance of Monitoring Models in Production

Models are trained based on historical data, and we expect them to work in the same fashion on new data. However, in the real world, things are constantly changing, and hence there is a need to monitor the model and keep it updated. Here are some of the reasons to monitor the models:

• Data drift can happen due to changes in input data with respect to training data.
• Concept drift can happen due to changes in model performance.
• Detect unfair bias introduced by the model.
• Changing business needs may require a model to be updated. For example, a retailer may introduce new product lines that require a product recommendation model to be retrained.
• New regulations ...