Large Language Models: The Hard Parts

Chapter 5. Structured Data Output

In limits, there is freedom. Creativity thrives within structure.

Julia B. Cameron

While LLMs excel at generating text, getting structured output that adheres to strict rules, like a binary “yes” or “no” or consistent JSON, remains difficult. The core problem stems from the way LLMs produce output token by token, with each token sampled from a probability distribution across a model’s entire vocabulary. Even if a model follows JSON formatting instructions with 99% reliability per token, this small chance of error compounds across all tokens in a response.

Traditional prompt engineering that asks for a JSON-formatted response is the lowest-cost solution to this problem, but it’s not perfect. Using a prompt to ask for JSON output puts constraints on an LLM, and the LLM might not always follow those constraints. This reliability issue becomes problematic when integrating LLMs into production applications. This has driven the development of constrained generation techniques (CGTs) that can guarantee structurally valid output.

CGTs solve this problem by changing where control is applied. Instead of prompting an LLM as part of the query, which is akin to giving stern instructions, CGTs intervene in the token sampling process itself. The mechanism works by first allowing the model to compute probability scores for all tokens in its vocabulary—typically around 50,000 options, depending on the tokenizer—then constraining which tokens would be valid. ...