The Meter Was Always Running

The first expensive agent run doesn’t look like a governance problem. It looks like a billing problem.

A team opens its first agent invoice after the meter turns on, sorts the runs by cost, and finds one that cost 40 times the median. The provider meter shows tokens and a total. The application logs say the request succeeded. The trace viewer shows a tidy request and a tidy response. None of them explain why this run wandered while its neighbors finished cleanly.

In my previous Radar article, “The Subsidy Ended: What Tool-Using Agents Actually Cost,” I argued that usage-based billing didn’t make agents expensive; it made their existing costs visible. The bill didn’t get bigger. It just got honest, and an honest bill is one you can engineer against.

But visible isn’t the same as attributable. To attribute cost in a tool-using agent, you have to see inside the run that produced it. Once you build that visibility, you discover that cost is only where the trouble first becomes visible.

Cost spikes, unsafe delegation, and runaway actions are different failures, but they expose the same missing layer: a control plane can’t govern a loop it can’t independently observe.

The bill is honest, but it isn’t explained

The number on the invoice isn’t wrong, only incomplete. Provider billing can tell you what was consumed; it usually can’t tell you which design choice inside your platform caused the consumption. Application logs can tell you whether the outer request succeeded; they often can’t tell you how the agent got there. That leaves teams arguing over a bill when the thing they need is an audit trail.

By control plane, I mean the platform layer above individual agents where an organization centralizes observability and enforces policy, access, budget, routing, and execution constraints. Most organizations have pieces of that layer already. What they often lack is the evidence layer underneath it: a loop-aware record of what the agent actually did, turn by turn.

The control plane is where policy decisions live. The observability substrate is the evidence the control plane reads from. The instrumentation points are the runtime chokepoints the agent can’t bypass: model gateways, tool proxies, API gateways, execution sandboxes, runtime harnesses, and policy engines.

Many organizations instrumented the application boundary, then deployed systems whose real work happens inside a loop. The result is a control plane with opinions but not enough evidence.

The loop is the unit of observation

Here’s the mistake underneath the empty trace. Agent observability is often treated as a heavier version of application observability, when it’s a different shape entirely. The unit of work changed, and the instrumentation didn’t. A traditional service handles a request and returns a response; the request is the natural unit you trace.

An agent doesn’t so much handle a request as work toward an outcome. It reasons, calls a tool, reads the result, reasons again, and continues until it decides it’s finished, hits a boundary, or escalates. A single user intent can fan out into many model calls, many tool calls, and a context window that changes on every turn. The signal that matters is the relationship between those turns, not only the timing of any one of them.

Three things follow from this, and each one breaks an assumption that application monitoring quietly depends on.

First, the context is accumulating state, not a fixed payload. Each turn may carry forward prior messages, tool descriptions, retrieved files, intermediate results, and earlier decisions. You have to be able to watch that state grow turn by turn, because the growth is where much of the cost and risk live.

Second, a tool call is a first-class decision, not an implementation detail. Which tool the model selected, what parameters it passed, how large the result was, and whether a policy constrained the call are all part of the governance record. Routing accuracy and routing cost are the same audit viewed from two directions.

Third, every run can become its own trace tree. The same prompt can take a different path on Tuesday than it took on Monday, so fixed call graphs and clean service maps assume a regularity the agent may not have. If the unit of observation is still the request, you will see 10,000 successful calls and never notice the one loop that ran 15 turns when it should have run three.

What the substrate has to capture

Once you accept that the loop is the unit, the requirement becomes concrete. You need a small, specific set of signals captured below the agent and stored where you can query across the whole fleet, not only inside a per-run viewer. In a pilot I’m running for a large healthcare organization, this is the layer we built first, on OpenTelemetry, Cloud Trace, and a usage-log table in the warehouse. The particular stack matters less than the shape, which generalizes well beyond it.

At minimum, each user intent should produce a run trace. Each loop turn should be represented as either a span or a stable grouping attribute. Model calls, tool executions, policy checks, retries, and postprocessing should be child spans or structured events beneath that turn. The exact naming convention isn’t as important as preserving the causal structure of the loop.

None of this is exotic, and the practical design work isn’t inventing new telemetry primitives but controlling cardinality, retention, payload capture, sampling policy, schema evolution, and the joins between trace data, usage data, identity data, and policy data.

The storage point is the part teams underestimate. If these signals land only in a tracing viewer, you can inspect one run beautifully and never reason about a thousand. Governance is a fleet question, not a single-trace question, so the substrate has to be queryable.

It also has to be designed with data minimization in mind: metadata by default, content capture by exception. Capturing a tool call doesn’t mean storing every raw prompt, full result set, credential, confidential document, or sensitive parameter in the trace. In regulated environments, the useful pattern is to separate metadata from payload: tool name, model, token counts, payload size, row counts, policy decision, authority context, request ID, and redacted or hashed parameter values where necessary. The goal is enough evidence to reconstruct why a run behaved the way it did, not an uncontrolled archive of everything the agent saw.

The first useful version doesn’t need full prompt capture or semantic evaluation. With columns like run_id, turn_id, parent_span_id, timestamp, principal_id, delegated_scope, model, input_tokens, output_tokens, cached_tokens, tool_name, result_bytes, row_count, policy_id, policy_decision, stop_reason, loop_bound_hit, and outcome_status, expensive loops stop being mysteries and start being queries.

The exact syntax will vary by warehouse, but the governance question should be expressible without a human clicking through individual trace viewers:

with runs as (
  select
    run_id,
    count(distinct turn_id) as turns,
    sum(input_tokens + output_tokens) as total_tokens,
    max(result_bytes) as largest_tool_result,
    bool_or(loop_bound_hit) as hit_loop_bound,
    count_if(policy_decision = 'rewrite') as rewritten_actions
  from agent_turn_events
  where occurred_at >= current_date - interval '7 days'
  group by run_id
)
select *
from runs
where turns > 10
   or largest_tool_result > 10000000
   or hit_loop_bound
   or rewritten_actions > 0;

That is the difference between admiring a trace and governing a fleet.

In the old trace, the expensive run from the opening was simply expensive. In the loop-aware trace, it becomes legible: turn 3 retrieved 80,000 rows, turn 4 carried that result forward, turn 5 selected the expensive model, turns 6 through 11 retried the same tool call with slightly different parameters, and the run finally stopped because it hit a loop bound rather than because it completed cleanly. The run stops being a riddle and becomes a record.

One substrate, three governance problems

The reason this is worth building once, properly, is that the same substrate answers the three agent governance problems that the industry often treats as separate: cost management, delegation and access control, and runaway-action prevention. They are not identical failures, but they require the same kind of evidence.

Cost is the first, and with token accounting on every turn you can finally answer why a run was expensive. You can see whether the cost came from too many turns, too much context carried forward, an oversized tool result, an expensive model used for the wrong step, or a retry loop that should have been bounded.

Delegation and access are the second, and harder, problem. In multi-agent systems, delegation is a security boundary. Enterprises will eventually be asked who authorized a given agent action, under whose authority it ran, and which data scope applied at the time. The audit trail for that question is this same trace, enriched with identity and authority on each turn.

Runaway actions are the third. The destructive delete that becomes a war story, the agent that tried to drop a production table, or the loop that repeatedly issued the same expensive scan shouldn’t only exist in a postmortem. In this model, the blocked destructive statement is a guardrail decision event with a deny on it, and the runaway scan is a trace that hit a loop bound or payload cap. The interesting governance signal is the dangerous action that a deterministic control refused.

Three conversations, one place to stand. The loop is the unit of governance because the loop is where cost accumulates, authority is exercised, tools are selected, controls fire, and outcomes emerge.

The agent can’t keep its own records

There’s a tempting shortcut to instrument the agent itself, to let the agent log its own tokens, its own authority, and its own blocked actions. That’s the fox keeping the henhouse ledger.

The agent can emit useful breadcrumbs, but it can’t be the system of record for its own authority, cost, or refusals. An agent reporting on its own scope and blocked actions is self-reporting, and self-reporting is exactly what fails an auditor and exactly what a clever prompt can talk its way around.

The substrate has to be instrumented below the agent, at the layer the agent can’t opt out of. In practice, below the agent means the model gateway, tool proxy, runtime harness, execution environment, API gateway, or policy engine: the layer the agent has to pass through, not a logger the agent can choose to call.

This is the through-line of the control-plane argument. The platform is where you enforce policy, access, budget, routing, and cost, and it can only enforce what it independently observed. Enforcement and observation are two faces of the same layer; put them anywhere the agent can edit, and you have neither.

We already have tracing, and it isn’t enough

The natural objection is that this is solved already: Mature tracing tools exist, agent observability vendors exist, and teams can turn on a trace viewer and see what happened. The gap isn’t visualization, since plenty of tools can show a useful trace of an agent run. The harder gap to cross is completeness and actionability: whether the trace carries the evidence a control plane needs, whether that evidence is independent of the agent, and whether it lands somewhere the organization can query across the fleet.

A useful test is whether the control plane can answer this without opening an individual trace viewer: Show me all runs this week where context grew by more than 5x, a tool returned more than 10 MB, a guardrail rewrote the action, and the run still reached a user-visible answer. If the answer requires a human clicking through traces one by one, you have visualization, not governance, and seeing one run isn’t the same as governing a thousand.

A dashboard tells you what happened. A control plane uses what happened to change what happens next, which requires the signal to live somewhere an enforcement decision can read it.

The pattern, not the stack

It would be a mistake to read this as an argument for a particular tracing standard, warehouse, vendor, or cloud platform. The stack is incidental; the shape is the point.

The recipe stays the same regardless: loop-aware traces; turns represented as spans, grouping attributes, or structured events; token, tool, guardrail, and identity evidence attached to those turns; storage you can query across the fleet; instrumentation that sits below the agent rather than inside it; and data minimization that keeps the trace useful without turning it into a shadow copy of sensitive payloads. Build it on whatever your platform already speaks.

The teams that treat observability as a dashboard will keep discovering their problems in the order the symptoms happen to surface: first as a surprising invoice, later as an audit finding, eventually as an incident. The teams that treat observability as the sensory layer of the control plane will see all three coming from the same data, and will be able to act before the meter, the auditor, or the incident forces the question.

Prompts guide behavior. Guardrails govern behavior. Observability is how you know the governance is real. You can’t govern what you can’t see, and you can’t improve what you can’t attribute.