Stop counting tokens, start counting what ships
Every AI coding demo ends the same way: a token counter ticks upward, someone jokes about the bill, and the room moves on. That counter is the wrong dashboard.
The MIT Technology Review story behind this piece makes the point sharply: when an agent gets a coding task, it does not answer once. It plans, writes code, runs tests, reads failures, and tries again. A single task can burn 10 to 50 times more tokens than a basic chat prompt. The more useful the agent becomes, the more calls it makes. That is not a bug in the economics of agents; it is the economics of agents.
So stop asking “how many tokens did we use?” Start asking “what shipped, and what did it cost to ship it?” Tokens are fuel. Merged pull requests, accepted migrations, fixed flaky tests, and closed tickets are output. A token budget is a leading indicator, not a scoreboard.

图示:来源页预览图(仅作编辑上下文;版权归属原发布方) · 来源引用:www.technologyreview.com
The metric that survives contact with an agent
Define a shipped unit before you compare tools. For most teams it is a merged PR that survives review and deployment, not an agent session that produced a diff. For platform work it might be an accepted migration, a dependency upgrade rolled out, or a test suite that now blocks regressions.
Then calculate cost per shipped unit:
(model + tool + CI + review + incident cost) / shipped units
That formula changes behavior. A task that uses 30x more tokens but replaces two hours of developer toil and passes tests may be cheap. A “efficient” agent that produces diffs humans rewrite is expensive. Track cost per merged PR by repository and by agent. Pair it with acceptance rate, rework rate, review minutes, CI minutes, and revert rate. If an agent’s PRs are merged but trigger incidents two weeks later, you have not shipped; you have deferred cost.
Roundup: where agents earn their tokens
1. IDE-native assistants: Cursor, GitHub Copilot, and similar
Use when: the work is local and scoped — a function rewrite, a test addition, a small refactor, an explanation of unfamiliar code. These tools shine when a developer stays in the loop and can accept or reject small diffs quickly.
Do not use when: you want long autonomous multi-file work, ambiguous product changes, or production firefighting. IDE assistants are optimized for interactive flow, not durable agent loops. They can still burn context and premium requests, but they are usually the lowest-risk entry point.
Pricing caveat: seat subscriptions are not API token pricing. Premium request quotas, model routing, and usage caps determine the real cost. Heavy agent-style use can hit limits that a light chat user never sees. Compare cost per accepted change, not sticker price per seat.
2. Terminal and repo-aware agents: Claude Code, OpenAI Codex CLI, Aider
Use when: the task is repo-aware and testable. Give the agent a reproduction, a failing test, a narrow migration, or a bug with a clear acceptance criterion. These tools can inspect files, run commands, and iterate against feedback — exactly the loop that consumes tokens.
Do not use when: requirements are vague, the test harness is weak, secrets are exposed, or the change has high blast radius. An agent without tests is a confident diff generator. Also avoid using them as a substitute for architecture review; they are good at local reasoning, not org-level trade-offs.
Pricing caveat: metered API usage, tool output, retries, and context compaction all affect the bill. Cache hits can make repeated work cheaper, but failed runs still cost. Measure cost per accepted diff, not cost per session. A long session that ends in a merged PR may be better value than three cheap sessions that produce nothing.
3. Async PR agents: GitHub Copilot coding agent, Devin, and similar
Use when: you have a backlog of well-specified chores — dependency bumps, test coverage, documentation fixes, small bug fixes, repetitive migrations. These agents can work asynchronously and in parallel, which is their real advantage.
Do not use when: the change touches authentication, payments, infrastructure, data deletion, or anything with a wide blast radius. Also avoid ambiguous product work that needs human discovery. Async agents amplify whatever specification and test coverage you already have — including its gaps.
Pricing caveat: these tools often price in credits, agent compute units, or seats plus concurrency. Failed runs, parallel attempts, and review cycles can multiply cost. The hidden cost is human review: an async agent that opens ten PRs can create a review queue that eats the savings. Pilot on a capped backlog with a clear definition of done.
4. Evaluation and observability: LangSmith, Braintrust, OpenTelemetry plus Grafana
Use when: you need to know whether agents are actually shipping. Instrument cost per PR, acceptance rate, rework rate, review time, and failure categories. This is the layer that turns token telemetry into unit economics.
Do not use when: you are building a vanity dashboard nobody checks. Start with one repo and five metrics. Also do not instrument everything on day one; event pricing and engineering time add up.
Pricing caveat: per-seat and per-event pricing can scale faster than model costs when you log every agent step. Sample traces, set retention limits, and decide which metrics drive decisions. Observability should pay for itself by preventing bad rollouts, not by producing prettier charts.
5. Local and self-hosted models: Ollama, vLLM, LM Studio
Use when: data residency matters, volume is predictable, and the task tolerates a quality gap — classification, simple edits, private code search, routine test generation. Local inference can be excellent for high-volume, low-complexity work.
Do not use when: you need frontier reasoning on hard bugs, or you lack GPU operations capacity. Idle GPUs are expensive, and model quality changes quickly enough that a local setup can become a maintenance project.
Pricing caveat: the bill is GPU capacity, utilization, and operations — not tokens. A local model that sits idle is not free. Compare fully loaded cost per shipped unit against API pricing, including the engineer who maintains it.

图示:HEXOPS 原创配图(受选题启发,非复制原文图片) · 来源引用:www.technologyreview.com
A pragmatic stack
A sensible default: IDE assistant for daily edits; a terminal agent for scoped, test-backed tickets; an async PR agent for a capped backlog of chores; an eval dashboard for cost per merged PR; and a local model only where privacy or volume justifies it. Keep a human review gate on anything that can page someone at 2 a.m.
The workflow matters more than the brand. Give agents tests, narrow tickets, and a definition of done. Route small tasks to cheaper models. Reserve frontier models for ambiguity and debugging. Cap parallel runs so review capacity does not become the bottleneck.
Pricing caveats that bite
Published token prices are not your effective cost. Caching, batch discounts, reasoning tokens, tool output, retries, and context compaction all move the number. Seat subscriptions hide usage caps. Credit-based agents can bill failed attempts. Vendor pricing and model routing change frequently, so re-check before you standardize.
The biggest hidden costs are human: review time, merge conflicts, security review, and incidents. An agent that ships a cheap PR which takes three engineers a day to debug is not cheap.
Stop optimizing the wrong number
Token usage is a useful operational signal — for capacity planning, rate limits, and spotting runaway loops. It is a terrible scoreboard. The scoreboard is shipped units per dollar, with quality and risk attached.
If an agent uses 10 to 50 times more tokens than chat but reliably turns a ticket into a reviewed, deployed change, that is not waste. That is the product working. Count what ships, then count what it cost to ship it.
Sources
- MIT Technology Review, “Stop counting tokens, start counting what ships”: https://www.technologyreview.com/2026/11/02/1147702/stop-counting-tokens-start-counting-what-ships/
- Vendor documentation for Cursor, GitHub Copilot, Claude Code, OpenAI Codex, Aider, Devin, LangSmith, Braintrust, Ollama, vLLM, and LM Studio. Pricing and product limits change; verify current terms before budgeting.



