Building Generative AI Services with FastAPI

Chapter 5. Achieving Concurrency in AI Workloads

Chapter Goals

In this chapter, you will learn about:

• The role and benefits of multithreading, multiprocessing, and asynchronous programming in enhancing AI application performance and scalability

• The role of the thread pool and event loop in a FastAPI server

• How to avoid blocking the server when processing requests

• Using asynchronous programming to interact with external systems such as databases, AI models, and web content by building a web scraper and a RAG module

• The model-serving strategies and LLM memory-optimization strategies to reduce memory-bound blocking operations

• Strategies for handling long-running AI inference tasks

In this chapter, you will learn more about the role and benefits of asynchronous programming in boosting the performance and scalability of your GenAI services. As part of this, you’ll learn to manage concurrent user interactions and interface with external systems such as databases, implement RAG, and read web pages to enrich the context of model prompts. You’ll acquire techniques for effectively dealing with I/O-bound and CPU-bound operations, especially when dealing with external services or handling long-running inference tasks.

We will also dive into strategies for efficiently handling long-running Generative AI inference tasks, including the use of FastAPI event loop for background tasks execution.

Optimizing GenAI Services for Multiple Users

AI workloads are computationally expensive ...