Large Language Models: The Hard Parts

Chapter 9. Epilogue: LLMBAs in the Era of Falling Costs

In Rome, everything has its price.

Juvenal

Throughout this book, we’ve covered the technical foundations of building, evaluating, and deploying LLMBAs responsibly—from choosing the right models and crafting effective prompts, to implementing structured outputs and RAG systems, to measuring performance with benchmarks like SALAD-Bench and TruthfulQA, to aligning models through DPO and protecting them with guardrails. These are the skills that let us build production LLMBAs today. But there’s a macro trend underlying everything we’ve discussed that will reshape what this work looks like tomorrow: inference costs are collapsing at an astonishing rate. When the a16z team analyzed this trend in mid-2024, it identified a 10x annual decline in costs. By early 2025, that rate had accelerated to 50–200x per year for equivalent performance levels.

This cost collapse changes the marginal cost of intelligence. Design decisions still involve cost-benefit trade-offs, but the balance shifts: evaluations that once felt too expensive to run broadly, guardrail checks that had to be used sparingly, and larger models that were difficult to justify economically become more feasible.

Implementing defense-in-depth with multiple guardrails becomes affordable. We can check inputs with Llama Guard and outputs with OpenAI Moderation and use an LLM judge for edge cases in a cost-effective manner. Using frontier models for evaluation and alignment ...