Retrieval-Augmented Generation in Production with Haystack

Chapter 4. Observable AI

Now that you’ve deployed your AI application, it is time to sit back, relax, and let users have a seamless experience with your application. Seamless because, after all, haven’t you evaluated your model offline on representative data and load-tested it prior to deployment in production? In many cases, however, performance in production varies and needs to be appropriately monitored to ensure the application behaves as expected. In traditional software applications, we care mostly about operational metrics (latency and throughput). But for AI applications, we also care about quality and performance.

Here’s an example of a case where performance is impacted. Let’s say, for example, that a product website builds a recommendation system. The performance is great initially, as customers find recommendations useful and sales go up. But a week later, performance starts to go down. It gets so bad that the new application is doing worse than the previous simplistic model. What happened? A few weeks of digging into the data shows that customers who bought a certain shoe were now given recommendations about the same shoe, just in a different color. Unfortunately, customers who just bought that shoe, giving careful consideration to their preferred color at the time of purchase, usually aren’t interested in buying another pair in a different color. By the time the issue is identified, it has resulted in weeks of lost time and frustrated customers—problems that could ...