Learning Ray

Chapter 8. Online Inference with Ray Serve

Edward Oakes

In Chapters 6 and 7 you learned how to use Ray to process data, train ML models, and apply them in a batch inference setting. However, many of the most exciting use cases for machine learning involve online inference.

Online inference is the process of using ML models to enhance API endpoints that users interact with directly or indirectly. This is important in situations where latency matters: you can’t simply apply models to data behind the scenes and serve the results. There are many real-world examples of use cases where online inference can provide a lot of value, for example:

Recommendation systems

Providing recommendations for products (e.g., online shopping) or content (e.g., social media) is a bread-and-butter use case for machine learning. While it’s possible to do this offline, recommendation systems often benefit from reacting to users’ preferences in real time. This requires performing online inference using recent behavior as a key feature.

Chat bots

Online services often have real-time chat windows to provide support to customers from the comfort of their keyboard. Traditionally, these chat windows were staffed by customer support staff, but a recent trend to reduce labor costs and improve time-to-resolution is replacing them with ML-powered chat bots that can be online 24/7. These chat bots require a sophisticated mix of multiple machine learning techniques and must be able to respond to customer input ...