Designing Deep Learning Systems

7 Model serving in practice

This chapter covers

• Building a sample predictor with the model service approach
• Building a sample service with TorchServe and the model server approach
• Touring popular open source model serving libraries and systems
• Explaining the production model release process
• Discussing postproduction model monitoring

In the previous chapter, we discussed the concept of model serving, as well as user scenarios and design patterns. In this chapter, we will focus on the actual implementation of these concepts in production.

As we’ve said, one of the challenges to implementing model serving nowadays is that we have too many possible ways of doing it. In addition to multiple black-box solutions, there are many options for customizing ...