What We Learned from a Year of Building with LLMs

Chapter 2. Operations: Developing and Managing LLM Applications and the Teams That Build Them

A possibly apocryphal quote attributed to many leaders reads: “Amateurs talk strategy and tactics. Professionals talk operations.” Where the tactical perspective sees a thicket of sui generis problems, the operational perspective sees a pattern of organizational dysfunction to repair. Where the strategic perspective sees an opportunity, the operational perspective sees a challenge worth rising to.

In Chapter 1, we introduced the tactical nuts and bolts of working with LLMs. In the next chapter, we will zoom out to cover the long-term strategic considerations. In this chapter, we discuss the operational aspects of building LLM applications that sit between strategy and tactics and bring rubber to meet roads.

Operating an LLM application raises some questions that are familiar from operating traditional software systems, often with a novel spin to keep things spicy. LLM applications also raise entirely new questions. We split these questions, and our answers, into four parts: data, models, product, and people.

For data, we answer: How and how often should you review LLM inputs and outputs? How do you measure and reduce test-prod skew? 

For models, we answer: How do you integrate language models into the rest of the stack? How should you think about versioning models and migrating between models and versions?

For product, we answer: When should design be involved in the application development ...