Skip to content
MUNEEB SHAFIQAll writing
Email

8 min readllm-cost · agents · evaluation · engineering-practice

Cheap by construction, unmeasured in practice

I designed two multi-agent systems to be cheap by construction, bounded batches, tiered model routing, grounding over cleverness. Then I went looking for what they actually cost, and found I had never recorded a single number. Here's what I found instead, and why it was the more interesting answer.

Muneeb Shafiq · Associate AI Engineer, Symufolk · Every claim below was checked against a primary source.

Over the last few months I built two multi-agent systems. One is a security scanner that grounds static-analysis findings in a corpus of real disclosed vulnerabilities before an LLM rules on them. The other is a nine-agent pipeline that produces course material behind human approval gates.

In both, I made deliberate decisions to keep LLM spend down. I was pleased with those decisions. Then I sat down to write about them, went looking for the numbers, and discovered I had never recorded a single one.

That turned out to be the more interesting story, so this is the one I'm writing.

Why you can't budget an agent from a price list

Start with the research, because it explains the shape of the problem.

Bai et al. ran eight frontier models through the OpenHands agent on all 500 SWE-bench Verified problems, four independent runs each. An agentic coding task consumed an average of 4.17 million tokens at $1.86. A multi-turn code chat about a coding problem: 3,390 tokens, about two cents. A single-turn code reasoning question: 1,190 tokens.

The gap is roughly three orders of magnitude, and the reason is not that the agent thinks harder. It's that the agent re-reads its own accumulated context at every step. The paper puts the input-to-output ratio for agentic work at about 154:1, against 1.33:1 for chat and 0.16:1 for single-turn reasoning. The ratio doesn't just grow, it inverts. In chat you mostly pay for what the model writes. In agentic work you almost entirely pay for what it re-reads.

Two more findings from the same paper matter for anyone trying to forecast a bill:

  • Repeated runs of the same model on the same task differed by up to 30× in total tokens. (The average spread between the cheapest and most expensive run is closer to 2×, 30× is the tail, not the typical case. But the tail is what blows a budget.) The extra spend bought nothing: accuracy rose modestly, then flattened, then degraded at the top cost quartile, with the most expensive runs showing markedly more repeated file views and re-edits.
  • Asked to estimate their own token cost before running a task, with permission to explore the repo first, models managed Pearson correlations between 0.04 and 0.39 against actual usage, and systematically underestimated. The model cannot tell you what the model will cost.

Meanwhile the unit price of AI capability is in freefall. Stanford HAI's 2025 AI Index tracked GPT-3.5-level performance on MMLU falling from $20.00 to $0.07 per million tokens in about 18 months, a 280-fold drop, and Epoch AI puts the rate of decline at anywhere from 9× to 900× per year depending on the benchmark.

And yet, per Menlo Ventures, enterprise spending on generative AI went from $11.5 billion in 2024 to $37 billion in 2025. Prices collapsed; bills tripled. Usage outran the discount.

Put those together and you get the operating conclusion: cost is a distribution, not a price, and you cannot derive it from a rate card. You either measure it or you don't know it.

So I designed for it.

What "cheap by construction" looked like

The scanner: bound the spend, then say what you skipped

The scanner's expensive step is LLM verification, taking each static-analysis finding, its surrounding code, and retrieved real-world exemplars, and asking a model for a verdict. Naively that's one API call per finding, and a large codebase produces hundreds.

Three decisions:

Batch, and cap. Verification runs at 12 findings per call, with a hard ceiling of 60 findings per scan. A scan costs a handful of requests rather than hundreds, which is what makes it survivable on a free-tier quota that is metered per day, per model.

Say what you skipped. Findings past the cap aren't dropped. They come back labelled Unverified, with a confidence of zero and an explanation that says so. The ranking puts them below everything the model actually looked at. This is the decision I'd defend hardest: a tool that quietly truncates its own output is worse than one that admits its budget, because the user can't tell the difference between "we checked and it's fine" and "we ran out of money."

Don't ground what you won't verify. Retrieval, embedding the code slice, querying the vector store, runs only for the findings that will actually reach the model. Grounding all of them meant paying retrieval cost for work that would never be used, and on a large codebase it stalled the scan outright.

The pipeline: route by where judgment concentrates

The nine-agent pipeline routes each agent to a model tier, and I wrote the rule down so it would survive me: Sonnet is the default. Opus is the exception, and each agent that holds it has to argue for it.

Three agents won that argument, the researcher who decides whether a course is viable at all, the architect who writes the syllabus every downstream artifact is built against, and the reviewer who runs the final 18-check pass. The justification is the same in each case: their errors are paid for in rebuild, not in tokens. A bad reviewer pass ships a broken package. A bad viability call spends the entire pipeline on a course that shouldn't exist. Next to that, the model price difference is noise.

Four agents run mid-tier, on the grounds that they execute against a contract that's already been approved rather than exercising open-ended judgment.

Two decisions I still like:

Tier and model are separate fields. The tier is the durable claim about how much judgment a role concentrates; the model ID is this quarter's answer to it. Change the model when the lineup changes; change the tier only when the role changes. The model IDs have already churned once. The tiers haven't moved.

Escalate the task, not the agent. Exactly one agent has a per-task escalation: the lab engineer runs on the large tier for one job, piloting the build before the gate, because that transcript is what caps how deep the whole module can claim to go, and it's the one failure the gate built to catch it can't see. Pulling the whole agent up a tier would have meant paying premium rates on all its bulk output to fix one task.

The highest-volume agent by message count runs on the cheapest model in the lineup, because its accuracy comes from a retrieval grounding pack rather than from raw model capability. That one has a real constraint attached: its 200K context window, against 1M for the rest, means the grounding pack has to fit, and a silently truncated pack is the failure mode to watch for.

Then I went looking for the numbers

There aren't any.

The scanner's LLM client never reads the token-usage field the API returns. There's no request counter, no cost calculation, no timing log. Not one committed scan result, benchmark, or fixture carries an observed value. Every number I have about that system is a constant I chose, 12, 60, not a measurement I took.

The pipeline is worse, because it knew better. Its own README instructs: record actual token spend per course from course one. The orchestrator's brief requires a running per-stage spend estimate in the status file. That table exists. Every row reads "not started" or "in progress, no estimate yet." The one course that ran was closed unfinished before a single figure was entered.

So I have two systems engineered for cost, and zero evidence about cost. The design is real. The validation never happened.

What was measured, by accident

Here's the part I didn't expect.

The pipeline keeps an append-only status file, state, gate approvals, decisions. It was specified at 120 lines maximum. It finished at 19,252 lines and 986 KB, across 310 separate commits to that one file. Roughly 160× its own spec.

Which would be a tidy story about scope creep, except for one detail: the review agent polls that file as its start condition. So the system's own audit log became a recurring input of roughly 193,000 tokens. The record broke its own reader. Nobody designed that; it accumulated.

Separately, one stage counted its own inbox and found 418 messages, 1,262,834 characters, about 316,000 tokens received. In a status update shortly before, the orchestrator had reported "roughly a hundred messages." It was off by a factor of five, about its own system, in writing, while looking at it.

Those are the only real token numbers either system produced. Neither is a model cost. Both are coordination cost, the price of agents keeping a shared record and talking to each other.

And that lands exactly where the research said it would. The dominant cost in agentic work is context volume, not model intelligence; input, not output; 154:1. I optimised the part I could see on a price list, which model, how many calls, and paid, unmeasured, for the part that actually dominates: context that grows on its own.

What I'd do differently

Instrument before optimising. Every provider returns token usage on every response. Reading that field and writing it to a row is an afternoon of work, and without it every efficiency claim is a belief. I had opinions about cost for months and evidence about it for none of them.

Meter the context, not just the calls. My cost controls all counted requests. The measurements that eventually surfaced were all about accumulated state. Any file an agent reads on every cycle is a recurring token cost, and it grows quietly, and a line-count cap in a spec is not enforcement.

Treat a self-reported estimate as a hypothesis. The orchestrator was wrong by 5× about its own message volume. The models in the study were wrong by more about their own token usage. Neither is lying; both are estimating without instrumentation. Same failure, different substrate.

Design for cost anyway. Batching, capping, tier routing, grounding over cleverness, I'd make all of those calls again, and the labelled-Unverified decision I'd defend in any room. Structural controls are worth having before you have data, because they bound the worst case. They just aren't evidence, and I should not have let myself confuse the two.

The honest summary is that I built the guardrails and skipped the gauges. The next number I publish about either system will be one I measured.

Sources