Hands-On LLM Serving and Optimization

Chapter 5. Challenges When Serving LLMs

So far in this book, we have demonstrated the core concepts of model serving, provided several architectural patterns for serving ML models, and analyzed the trade-offs involved in deploying models at scale. By now, we hope you have a strong understanding of model serving paradigms, because we are about to take a significant step into a different realm. In this chapter, we will shift our focus to one of the fastest-growing fields in the AI world: optimizing LLMs for serving.

Since the rise of ChatGPT in late 2022, LLMs have transformed how AI is applied in real-world scenarios, from chatbots and code generation to advanced reasoning and decision-making systems. However, their sheer size, computational demands, and unique serving requirements introduce challenges that can go far beyond classic model serving techniques. From novel ideas to widely adopted frameworks, the field of optimizing LLMs for faster and more efficient serving performance has evolved at an unprecedented pace. It can be daunting: anyone not familiar with this area can easily get overwhelmed. For example:

• When reading technical blogs, you may wonder: “What is this vLLM framework that has gained popularity in just a year or two and has already been adopted in so many places?”

• When reading research papers, you may ask: “How does FlashAttention work, and how can I optimize it at the hardware level to speed up LLM inference?”

• When following AI news, you may come across ...