Machine Learning Production Systems

Chapter 11. Introduction to Model Serving

This chapter discusses model serving—the use of a trained model to generate predictions or results. Also referred to as running inference, model serving is the ultimate goal of any trained model.

Training a good ML model is only the first part of the production ML journey. You also need to make your model available to end users or to the business processes that rely on your model’s results. Serving it, or including it in an application, is how you make your model available.

Note

In the ML space, the words prediction, result, and inference are used somewhat interchangeably.

Model Training

In general, there are two basic types of model training:

Offline training

The model is trained on a set of already collected data. After deploying to the production environment, the model remains frozen until it is retrained with new data. The vast majority of model training is offline.

Online training

The model is regularly being updated as new data arrives (e.g., as data streams). This approach is generally limited to cases that use time series data, such as sensor data or stock trading data, to accommodate rapid changes in the data and/or labels. Online training is fairly uncommon and requires unique modeling techniques.

Model Prediction

In general, there are two basic types of model predictions:

Batch predictions

The deployed model makes a set of predictions based on a batch input data containing multiple examples. This is often used when ...