Preface
In early 2024, a team I worked with at a large fintech deployed a customer support agent powered by Claude Sonnet. The proof-of-concept was brilliant — it resolved tickets faster, customers loved it, and leadership greenlit a production rollout. Nobody checked the cost model. Within six weeks the monthly API bill crossed six figures. The agent was calling tools in recursive loops, stuffing entire conversation histories into every request, and nobody had instrumented a single token counter. We killed the project on a Thursday afternoon.
That experience radicalized me about AI cost engineering. Not because the technology was wrong — it was working beautifully — but because nobody on the team had a mental model for how LLM costs actually ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access