Chapter 1. The Economics and Anatomy of Production AI Inference
Imagine the board has a clean AI mandate: reduce external AI spending, host our own 300B-parameter model for workloads that make sense internally, and stop treating external AI spend as an unmanaged default. The work is not a rejection of SaaS; it is a workload-by-workload decision about what should stay external and what should move inside. On a slide it looks tidy. One model candidate. One platform plan. A smaller external bill. The diagram is calm, which is often the first warning sign.
When we see a plan like this, we want the room to pause on a different question: what task are we trying to complete, and what counts as success?
The first product built on that mandate is a unified internal assistant. A support engineer asks whether an application can be deployed in a particular region without violating data residency rules. Ten minutes later, a developer asks why a build failed after a dependency upgrade. To the board, both interactions may look like tokens used to leave the company that ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access