Machine Learning Model Serving Patterns and Best Practices

5

Keyed Prediction

In this chapter, we will discuss keyed prediction, a pattern of model serving that involves passing a key with every training instance to facilitate an easier mapping of input to output during the collection of responses. When we need to make a large batch of predictions in the same call and multiple parallel computing servers are assigned to predict a subset of the batch, then we might lose the order of prediction. This will cause problems in mapping the prediction to the features. To solve this problem in distributed or multi-threaded serving, we pass a key along with the feature. The servers add this key along with the prediction. We can use this key to map features to predictions. In this chapter, we will discuss in detail ...