What Is Agent Run Metrics? The Beginner's Guide to the JSON Format for Counting What an Agent Run Costs

CodexFuse Team · 2026-11-05T20:17:10.425Z · 27 min read min read

What Is Agent Run Metrics? The Beginner's Guide to the JSON Format for Counting What an Agent Run Costs

What Is Agent Run Metrics?

Agent Run Metrics is a JSON interchange format that describes the resource consumption of a single run of a language-model agent, step by step. It was proposed as Internet-Draft draft-arsentev-agent-run-metrics-02 by Evgenii Arsentev on November 1, 2026. The format carries counters, timing, and cost attribution only, and it deliberately excludes prompt and completion content.

The problem it solves is simple to state and hard to measure. A coding agent re-submits its whole accumulated context to the model on every step, so the same tokens get paid for again and again.

Each vendor reports that consumption in its own shape, which makes runs from different runtimes impossible to compare. Agent Run Metrics gives every runtime the same JSON shape for the same facts.

If you run agents in production, you already care about the inputs to this problem. You verify downloaded models with a SHA-256 hash generator before you trust them, and you rent the servers they run on. Most teams run their agents on VPS hosting with enough memory for long contexts, not on a laptop, because a multi-step run can accumulate tens of thousands of input tokens before it finishes.

This guide explains the agent run metrics JSON format in plain terms: what a report contains, the one-step-one-invocation rule that keeps counts honest, the five derived quantities the draft defines exactly, and what the format deliberately leaves out.

Why Does an Agent Run Need Its Own Accounting?

Per-request accounting is not enough for agents, because an agent run is not one request. A run is a loop: the agent reads a result, adds it to the context, and calls the model again with everything it has seen so far. The context grows with every step, and the input submitted at step ten generally contains the input submitted at step nine.

That repetition is where the money goes. Total input volume grows faster than the number of steps, even when each step adds little new material.

Content admitted early in a run is re-submitted on every later step, so where content enters a run matters as much as how big it is. A 600-kilobyte document fetched in step one costs far more over a long run than the same document fetched in the last step.

Agent run metrics vs token usage is the wrong comparison to make, because they answer different questions. Token usage tells you what one model call consumed. Agent Run Metrics tells you what the whole run consumed, which steps consumed it, and how much of it was the same context sent again.

A team that only watches per-call token counts can see every call look cheap while the run quietly burns through its budget. That is why agent run cost accounting starts from the run, not the call.

The draft's evidence comes from one practitioner's corpus, and it is worth quoting with that limit attached. The author analysed the execution journals of one agent runtime covering 722 runs, 150,902 model invocations, and about 34.6 billion tokens.

That works out to about 209 invocations per run and about 229,300 tokens per invocation. Those magnitudes come from a single runtime and a single provider, and the draft itself says no claim is made that they generalise to other deployments.

Two findings from that corpus explain why the draft exists at all. Summing per-block journal records overstated total consumption by a factor of about 1.90, while taking only the first record of each response understated output by a factor of about 1.8.

Both errors are silent, because the resulting report still looks internally consistent. The one-step rule in the next sections exists to stop exactly that.

What Does an Agent Run Metrics Report Contain?

A report is one JSON document describing one identified run. It is produced by the agent runtime, which the draft calls the Reporter, and consumed by cost-management, capacity-planning, research, and audit tooling, which it calls the Collector. Aggregation across runs is the consumer's job, not the format's.

The report is deliberately small. It is not a time-series metrics protocol and not a distributed tracing system, although it can reference W3C Trace Context identifiers when a tracing pipeline exists. Every member has a fixed meaning, and a Collector must ignore members it does not recognise, so the format can grow without breaking old readers.

The run object carries the identity of the run and its totals. The table below lists its members as the draft defines them, with the presence rules that matter most: what is required, what is optional, and what is conditional on the run's state.

The smallest conforming report needs only six members: version, run_id, start, end, status, step_count, and totals. The draft's own minimal example is a three-step run that finished in about 74 seconds, with totals of 41,200 input tokens and 1,860 output tokens. That example is an illustrative construction, and the draft states plainly that none of its values were measured from any system.

Two design choices deserve attention. First, timestamps must be RFC 3339 date-time strings using the Z offset, and local offsets are excluded because they would disclose the operator's location. Second, there is no separate duration field anywhere in the format: durations are derived from the start and end timestamps, so a report can never disagree with itself about how long something took.

What Counters Does Each Step Carry?

Each step carries a usage object with token counters, and the draft is unusually precise about what each counter includes. The two required counters are input_tokens and output_tokens, and both are defined to include everything, not just the billable part.

Input tokens are all tokens submitted as input, whether or not they were served from a cached prefix. Output tokens are all tokens the model produced, including internal reasoning tokens that were never returned to the caller.

The optional counters split those totals into the pieces that actually drive cost. Providers price the three ways an input token can be handled differently, commonly charging a premium for tokens stored into cache state and a discount for tokens served from it. Two runs with identical input_tokens can therefore differ in cost by a large factor, which is why the draft treats cache counters as first-class data rather than a footnote.

Three invariants hold the counters together. Cache read and cache write tokens are disjoint subsets of input tokens, and their sum must never exceed input_tokens.

When the steps array is complete, totals must be the member-by-member sum of every step's usage. And a reporter with no metering information must omit the billable members rather than copy the physical counts into them, because a guessed billable figure is worse than an absent one.

The cost object follows the same honest-labelling discipline. It carries a three-letter ISO 4217 currency code, an amount written as a JSON string rather than a number, and a basis of metered, list_price, or estimated. The amount is a string so the value survives parsers that map numbers to binary floating point, and a collector must not convert it to floating point before comparison or summation.

A reporter must not claim the metered basis for a figure it computed itself. Cost figures are attributions made by the reporter, and the draft states they must not be treated as authoritative for settlement.

What Is the One-Step-One-Invocation Rule?

One reported step equals one completed model invocation. That is the single most important sentence in the draft, and it is stated as a normative rule: a step of kind model_invocation must describe exactly one completed model invocation, and a report must not contain more than one step describing the same invocation. Everything else in the format exists to make that sentence checkable.

The rule matters because runtimes do not naturally produce one record per invocation. A response can arrive incrementally, in several content blocks, or as several journal records while it is still being produced.

Without a rule, a collector cannot tell whether five records mean five calls or one call written down five times. The agent run metrics draft resolves the ambiguity in the reporter's favour only in one direction: group, never split, never sum snapshots.

1. Group every record that belongs to one invocation into one step, using an identifier of the invocation, never timestamp, model identifier, or adjacency.
2. Report the output_tokens of the whole completed response, never a snapshot taken before the response ended and never the sum of snapshots observed while it was produced.
3. Count reasoning blocks inside the same step through reasoning_tokens, and count tool-invocation requests inside its output_tokens, while reporting the tool calls themselves as separate steps of kind tool_call.
4. Report a retried attempt as a distinct step with its own index and usage, linked by retry_of to the first attempt, and never merge attempts into one step with summed counters.
5. Never emit a step for a partial or intermediate usage structure observed while a response is still being produced, and never include its counters in totals.

An invocation that never completed may still be reported as a step carrying an error object plus whatever counters the provider metered. A step of kind model_invocation should carry an invocation_id, and a collector receiving two steps with the same invocation_id in one run must treat the report as malformed. That hard failure is the point: a double-counted invocation should break the report loudly instead of inflating a bill quietly.

The draft's own corpus shows what happens without the rule. Summing per-block journal records overstated total consumption by a factor of about 1.90, while taking only the first record understated output by a factor of about 1.8. In a corpus where slightly more than half of all model invocations happened inside delegated runs, those errors compound across the whole delegation tree.

How Do You Read the Five Derived Quantities?

The derived object holds five quantities, and the draft specifies each computation exactly. A reporter may omit the derived object entirely, but any member it includes must be computed exactly as specified, and reporters should not emit more than six significant digits. All five are per-run: a parent run's values say nothing about its children, and tree values cannot be combined.

Input amplification is the quantity the whole format is built around. A run whose context grows steadily has an amplification of about half its step count, a run whose context is fully rebuilt at each step has one close to its step count, and a single-step run has exactly 1.

It deliberately uses the largest context reached, not the final context, so runs that compact or truncate their context stay comparable. In the author's corpus the median was about 24, the 90th percentile about 125, and the largest observed value was over 4,000.

A worked example makes the arithmetic concrete. Take a three-step run whose model-invocation steps submit 12,000, then 24,000, then 36,000 input tokens. The sum is 72,000 and the largest single step is 36,000, so the input amplification is 72,000 divided by 36,000, which equals 2.0.

If 30,000 of those 72,000 input tokens were served from cache, the cache hit ratio is 30,000 divided by 72,000, or about 0.4167. If the run produced 3,600 output tokens, the output share is 3,600 divided by 75,600, or about 0.0476, which means less than 5 percent of all tokens in the run became output.

The cache ratios carry a cost warning that is easy to miss. In the author's corpus, tokens served from a cached prefix accounted for about 56 percent of attributed cost and tokens stored into cache state for about 32 percent, leaving about 12 percent for input processed afresh and all output together.

Storing into cache was the second-largest component of cost rather than a marginal one. A reporter that omits the cache counters leaves the greater part of a run's cost unattributable, which is why the draft treats their absence as a real gap instead of a minor omission.

One caution belongs here, because the draft contradicts itself on this figure. Section 3.4 says tokens stored into cache state accounted for about 27 percent of attributed cost, while Section 12 and the change log say the figure was corrected from about 27 to about 32 percent.

The 32 percent figure is the corrected one, and the 27 percent in Section 3.4 appears not to have been updated in this revision. This guide uses 32 percent throughout.

How Does the Format Handle Delegated Runs Without Double Counting?

Agents delegate. A parent run hands a subtask to a sub-agent, which runs its own steps, spends its own tokens, and reports its own run.

The question the draft answers is who owns those tokens: the child, the parent, or both. Its answer is the child, with the parent allowed to summarise, and never both at once.

A delegated run is a run in its own right, with its own run_id, steps, totals, and report. The relationship is expressed in three places: the child report's parent_run_id, the child report's root_run_id, and a step of kind delegation in the parent, spanning the interval the child ran and carrying the child's run_id where the parent knows it.

Those three members are accounting references only. The draft states they say where the consumption of a run is attributed and nothing else, and they are not evidence that the parent authorised the child.

The collector's rule is the one that prevents double counting. It must use exactly one of two methods for a tree: sum the totals of every run in the tree, or take the root's subtree_totals. It must not combine them, and in particular it must not add a run's subtree_totals to the totals of that run's descendants.

Either mistake counts the delegated work twice. The draft adds a reporting duty for the gap case: a known delegated run whose report has not been received is missing, not zero, and a collector must mark any figure derived from the partial sum as incomplete and should list the missing run identifiers.

The draft's aggregation findings give collectors a vocabulary for what can go wrong with a tree: complete, missing-report, subtree-mismatch, malformed-report, namespace-violation, and unauthorized-attachment. The root's subtree_totals is checked per counter and reported as consistent, inconsistent, or unchecked. In the author's corpus slightly more than half of all model invocations occurred in delegated runs, so ignoring delegation would describe a minority of the work.

What Does Agent Run Metrics Deliberately Leave Out?

The format's most important feature may be what it refuses to carry. A report must not contain any part of the context, of a model response, of a tool's arguments, or of a tool's result. That prohibition is absolute, and it applies to every member of every object defined in the draft, including free-form members such as labels, the agent object's name, the tool object's name, and the error object's type.

The reason is about audiences. A metrics pipeline typically has a wider audience, longer retention, and weaker access control than the system that produced the run.

Counters can travel to finance, planning, and audit teams who should never see the prompts behind them. Keeping content out of the format entirely is stronger than redacting it later, because there is nothing to leak if a report is forwarded, stored too long, or read by the wrong team.

That discipline extends to the other exclusions. The format defines no protocol between an agent and a language model, and none between an agent and its tools. It defines no pricing, billing, or settlement mechanism, and its cost fields are attributions rather than invoices.

It is not a distributed tracing system, although it can reference trace identifiers. And it carries no semantic content of a run at all: prompts, completions, tool arguments, and tool results are out of scope by design.

The step kinds show the same restraint. There are six: model_invocation, tool_call, retrieval, delegation, human_input, and compaction.

A tool call step carries a tool object with a required name and outcome, but the name must name the tool class within the reporter's scope and must not encode arguments, targets, identifiers, or any content of the call. An error object carries a short lowercase type token and an optional retryable flag, never a provider's free-text error message, because that message may quote the context.

How Do Reports Travel From Runtime to Collector?

A reporter may emit more than one report for the same run, and the draft designs that flow so a collector never has to accumulate. Every report is self-contained: its totals cover the run from its start up to the moment of the report, so a collector replaces rather than adds.

A revision number orders out-of-order arrivals, and a collector must discard a report whose revision is not greater than one already accepted. At most one report per run_id may carry a terminal status.

The optional HTTP binding is deliberately thin. A reporter posts to a collector URI configured out of band, with a content type of application/json, or application/json-seq for a sequence of reports. The request must travel over a connection providing confidentiality and integrity, which in practice means TLS.

A collector should respond with 202 Accepted and an empty body, or with a 4xx problem-details document on rejection, and 413 when a size limit is exceeded. Delivery is idempotent by construction on the pair of run_id and revision, and retries should use exponential backoff with jitter.

Two details protect the collector from abuse. First, the format defines no dedicated media type, and the requests for a standards-tree media type and two IANA registries are explicitly deferred until a working group decides whether to adopt the document.

Second, a collector must enforce limits on report size, the number of steps processed, and the number of distinct run_id values tracked per reporter, and should apply per-reporter rate limits. Where reports drive chargeback, quota enforcement, or capacity decisions, a reporter has an incentive to under-report, and a neighbour in an adversarial multi-tenant setting has an incentive to over-report on another tenant's behalf.

That incentive problem is why the draft is blunt about trust. Every value in a report is asserted by the reporter, and the format provides no means for a collector to verify a count, timestamp, or cost against the provider. A collector must not treat a report as evidence of what a run consumed; it is a statement by the reporter about what the run consumed.

A signature binds bytes to a signer but does not verify values, and independent guarantees must come from reconciliation against provider metering or independent measurement. The optional evidence_refs member lists URIs of outside records, but its presence is not evidence, and a collector must not dereference them automatically.

What Can the Counters Reveal About Your Prompts?

Counters are not content, but they leak information about it, and the draft's privacy section is candid about how. The length of a document is closely related to the increase in input_tokens at the step that first submitted it. A tool_call step name discloses which class of tool was used.

Timing members disclose when a run executed and, in aggregate, an operator's working hours. Where inputs come from a small or guessable set, an observer can compare token lengths against a report and identify which input was used.

The draft's advice follows from that leakage. A reporter should treat reports as confidential to the same degree as the workload they describe, and should not publish per-step reports for runs whose inputs are sensitive.

Publishing only run-level totals and derived quantities substantially reduces, but does not eliminate, the exposure. Operators should apply retention limits and aggregate once individual runs are no longer operationally relevant.

Identifiers carry their own rules. A run_id must not be derived from an identifier of a natural person, a session, an account, or a request the collector could correlate with one; reporters should generate unstructured values such as UUIDs.

Labels must not carry content from the run and should not carry identifiers of natural persons. Retaining reports is processing of data about a person's activity even though no personal data appears in the report, which is why the retention guidance exists at all.

Delegation adds a security rule of its own. A collector must not allow a reporter to attach a run to a parent outside that reporter's namespace without an authorisation decision of its own, because otherwise one party could inflate another's accounted consumption. Reporters must be authenticated, and a collector must reject a report claiming a run_id outside the authenticated reporter's namespace.

How Does Agent Run Metrics Relate to IoA Usage Accounting?

Agent Run Metrics does not stand alone. Two other drafts reference it, and together they sketch a small family of agent-accounting work that teams building agent infrastructure should watch as one group rather than three isolated documents.

The closest sibling is Verifiable Usage Accounting for the Internet of Agents, draft-zhang-ioa-usage-accounting-01, dated November 2, 2026. It defines a common object model for verifiable usage accounting across domains, with accounting contexts, metering profiles, measurement dimensions, usage event records, and accounting evidence references. Its relationship section states that Agent Run Metrics can supply source measurements for that framework, and that a mapping must preserve its counting conventions, local versus subtree scope, reporter namespace, and missing-report status.

It claims no lossless or implemented mapping, and it does not mandate the Agent Run Metrics format. The two drafts answer different questions: this one counts one run, that one accounts for collaboration across domains.

The other reference comes from the same author. The LLM Context File Discovery draft cites Agent Run Metrics for the retransmission-cost effect, noting that material admitted once is paid for on every subsequent step of a run, so the cost of retrieving a 600-kilobyte detail resource is not the transfer but its persistence in the context window. That single sentence is the economic argument for the whole format, and it is also why discovery of context files matters: what you let into a run, you pay for repeatedly.

The wider agent-framework family gives this draft its context. The I2ICF work on integrated agent frameworks, described in What the I2ICF framework is, covers how agents are organised and managed, while What the Agentic AI Security Framework is covers how agents secure the networks they run on.

Agent Run Metrics covers what those agents cost. A team building on MCP over MoQ for agent communication will recognise the same pattern: a small, exact format that lets independent implementations compare notes instead of each inventing its own.

Who Should Follow This Draft, and What Comes Next?

The agent run metrics IETF draft is in its third revision, and it is still young. The -00 appeared on September 10, 2026, the -01 followed on October 5, and the -02 arrived on November 1, 27 days after the -01 and 52 days after the -00.

That is a live, maintained draft, and the -02 moved it from the Independent Submission stream to the IETF stream with intended status Standards Track, for discussion on the audit mailing list. Intended status is intent, not a granted status: the draft is an individual submission, not adopted by any working group.

Readers should also know what does not exist yet. The draft states that no implementation emits or consumes complete reports, and that no reference emitter or validator exists at the time of posting.

The partial implementations it names are counters-only analysis scanners published with the author's Context Economy report, and contextburn, the author's own command-line tool and MCP server that reads local Claude Code transcripts and reports what share of paid tokens became output, by token count and weighted by price. Contextburn covers derived quantities only and does not emit reports in this format.

The draft expires on May 5, 2027, which is 185 days after the -02 was posted. If it is not revised by then, it lapses.

The next revision to watch for would add the machine-readable schema the draft says is expected in a future revision, and possibly a reference emitter now that the format has moved to the IETF stream. Until then, the -02 is the document to read, and the counting rules in it are already usable: one step per invocation, totals that sum from steps, cache counters kept separate, and delegated work counted once.

Sources

Arsentev, E. "Agent Run Metrics: A JSON Interchange Format for Resource Accounting of Language-Model Agent Runs." Internet-Draft draft-arsentev-agent-run-metrics-02, November 1, 2026. https://datatracker.ietf.org/doc/html/draft-arsentev-agent-run-metrics-02

IETF Datatracker. "History for draft-arsentev-agent-run-metrics." Revision history confirming -00 on September 10, 2026, -01 on October 5, 2026, and -02 on November 1, 2026. https://datatracker.ietf.org/doc/draft-arsentev-agent-run-metrics/history/

Zhang, L., et al. "Verifiable Usage Accounting for Internet of Agents." Internet-Draft draft-zhang-ioa-usage-accounting-01, November 2, 2026. https://datatracker.ietf.org/doc/html/draft-zhang-ioa-usage-accounting-01

Arsentev, E. "LLM Context File Discovery." Internet-Draft draft-arsentev-llm-context-discovery-01, October 2026. https://datatracker.ietf.org/doc/html/draft-arsentev-llm-context-discovery-01

Arsentev, E. "contextburn: a run-efficiency meter for language-model coding agents." Version 0.2.1, September 2026. https://github.com/arsentev-ai/contextburn

IETF Datatracker. BibTeX record for draft-arsentev-agent-run-metrics-02, confirming 45 pages and November 2026. https://datatracker.ietf.org/doc/draft-arsentev-agent-run-metrics/02/bibtex/

Frequently Asked Questions

What is agent run metrics?

Agent Run Metrics is a JSON interchange format that describes the resource consumption of a single language-model agent run and its steps. It carries counters, timing, and cost attribution only, and it deliberately excludes prompt and completion content. It was proposed as Internet-Draft draft-arsentev-agent-run-metrics-02 in November 2026.

Is Agent Run Metrics an adopted standard?

No. It is an Internet-Draft, which is a work in progress, not an adopted standard. The -02 moved to the IETF stream with intended status Standards Track, but it is an individual submission that no working group has adopted, and it expires on May 5, 2027 unless revised.

What does one step equal one invocation mean?

It means a step of kind model_invocation must describe exactly one completed model invocation, and a report must not contain two steps describing the same invocation. A response delivered in several blocks or journal records is one invocation, retries are separate steps, and partial records observed while a response is still being produced are never metrics.

Does an Agent Run Metrics report contain my prompts?

No, and it must not. The draft's prohibition on prompt and completion content is absolute and applies to every member of every object, including labels, tool names, and error types. Counters can still leak information about content, such as document lengths, so sensitive runs should be reported at run level only.

How is Agent Run Metrics different from token usage?

Token usage counts what one model call consumed. Agent Run Metrics counts what a whole run consumed, step by step, including how much of it was the same context re-submitted on every step. Its derived quantities, such as input amplification and cache hit ratio, describe run behaviour that per-call token counts cannot show.