Building Generative AI Services with FastAPI

Chapter 6. Real-Time Communication with Generative Models

Chapter Goals

In this chapter, you will learn about:

• When to implement real-time communication in AI workflows

• Web communication mechanisms by comparing their features, differences, and similarities

• Real-time communication mechanisms including server-sent events (SSE) and WebSocket (WS)

• Selecting the correct streaming technology for your use case

• Implementing mocked streaming endpoints from scratch for testing and prototyping

• Implementing real-time API endpoints with both SSE and WebSocket mechanisms

• How to gracefully handle exceptions and close streaming connections

• API design patterns to simplify streaming endpoints

This chapter will explore AI streaming workloads such as chatbots, detailing the use of real-time communication technologies like SSE and WebSocket. You will learn the difference between these technologies and how to implement model streaming by building endpoints for real-time text-to-text interactions.

Web Communication Mechanisms

In the previous chapter, you learned about implementing concurrency in AI workflows by leveraging asynchronous programming, background tasks, and continuous batching. With concurrency, your services become more resilient to matching increased demand when multiple users access your application simultaneously. Concurrency solves the problem of allowing simultaneous users to access your service and helps to decrease the waiting times, yet AI data generation remains ...