Building Generative AI Services with FastAPI

Chapter 5. Achieving Concurrency in AI Workloads

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book. The GitHub repo is available at https://github.com/Ali-Parandeh/building-generative-ai-services.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

Chapter Goals

• Understand the role and benefits of multithreading, multiprocessing and asynchronous programming in enhancing AI application performance and scalability

• Understand the role of the thread pool and event loop in a FastAPI server

• Learn to avoid blocking the server when processing requests

• Use asynchronous programming to interact with external systems such as databases, AI models, and web content by building a web scraper and a Retrieval Augmented Generation (RAG) module

• Learn the model serving and LLM memory optimization strategies to reduce memory-bound blocking operations

• Discover strategies for handling long-running AI inference tasks

In this chapter, you will learn more about the role and benefits of asynchronous programming in boosting the performance and scalability of your GenAI services. As part of this, you’ll ...