Designing Deep Learning Systems

6 Model serving design

This chapter covers

• Defining model serving
• Common model serving challenges and approaches
• Designing model serving systems for different user scenarios

Model serving is the process of executing a model with user input data. Among all the activities in a deep learning system, model serving is the closest to the end customers. After all the hard work of dataset preparation, training algorithm development, hyperparameter tuning, and testing results in models is completed, these models are presented to customers by model serving services.

Take speech translation as an example. After training a sequence-to-sequence model for voice translation, the team is ready to present it to the world. For people to use this model remotely, ...