AISI says agent benchmarks need compute curves, not static scores

Why it matters

Agent startups increasingly sell outcomes, not chat. If benchmarks omit the token budget behind those outcomes, customers and regulators are comparing products on incomplete evidence.

Jessica McFadyen, Ole Jorgensen, Harry Coppock, Kevin Wei and Cozmin Ududec have put a disclosure problem at the center of AI agent evaluation: a benchmark score means far less if it does not say how much inference compute the agent was allowed to spend.

The UK AI Security Institute, a government research group inside the Department for Science, Innovation and Technology, published the argument on July 2 in a blog post summarizing its June 16 arXiv paper, "How Inference Compute Shapes Frontier LLM Evaluation". The work was amplified over the weekend by Toby Ord and by Ian Hogarth, AISI's chair, in posts surfaced by Aligned News.

Toby Ord on X

Ian Hogarth on X

The claim is narrow and important. AISI is saying that a single pass rate, fixed at a single token cap, can misstate what an agent can do. Raise the budget and the same system may finish harder work, recover from mistakes, or win through repeated attempts. Keep the budget low and a failure may measure the test protocol rather than the model.

That creates an immediate problem for model labs, benchmark vendors and startups selling agentic software. A static score is easy to market. A curve is harder to compress into a sales slide. AISI's work argues that the curve is the actual measurement.

A score with a hidden budget

AISI's Science of Evaluation team swept token budgets across cyber, software engineering, math, academic and medical benchmarks, rather than scoring each model once at a fixed cap. The group tracked how success, reliability, efficiency and task reach changed as models received more test-time compute.

The paper tested multiple frontier language models across several benchmarks, using larger token budgets, context compaction and repeated submission attempts. Its abstract makes the protocol point directly: benchmark scores are sensitive to inference compute, and evaluations should report capability as a function of that compute, especially in safety and policy settings.

AISI's public numbers show why the shift matters. On its cyber capture-the-flag suite, about 8% of tasks were solved only after the budget reached more than 10 million tokens, with some requiring up to 50 million tokens. On public software-engineering tasks, increasing total token budgets from 1 million to 10 million tokens raised performance by about 25%, according to AISI. On math and academic tasks, performance rose about 22% up to 5 million tokens.

The details cut against a common benchmarking habit: treating a low score as evidence that a model cannot do the work. In AISI's framing, some low scores are censored observations. The run ended before the agent had enough room to search, test, revise or try again.

AISI's analysis also found limits: some tasks plateau within typical budgets. The result is not a blanket claim that tokens always buy competence; they buy headroom when tasks provide feedback that agents can exploit.

The ranking can move when the budget moves

The sharper finding is comparative. AISI says newer models benefited more from additional test-time compute than older models. That means a fixed-budget evaluation can shrink the apparent gap between model generations, while a larger budget can widen it.

For anyone buying, building or regulating agents, that changes how benchmark tables should be read. A model that looks only modestly ahead at 1 million tokens may pull away at 10 million. A system that appears safer or less capable under one cap may look different when the cap rises. AISI's recommendation is to compare models across a shared compute range at matched budgets, rather than letting each headline number stand alone.

The same issue appears in time-horizon estimates, the metric used to describe how long a task an AI agent can complete relative to a skilled human. AISI previously estimated that frontier cyber time horizons on its CTF suite doubled every few months when measured at 2.5 million tokens per task. In the new analysis, horizons measured at 50 million tokens were far longer than at 2.5 million, changing the fitted trend.

A concrete example makes the abstraction less slippery. AISI says one recent frontier model's cyber horizon rose from around 40 minutes at a 2.5 million-token budget to around 4 hours at 50 million tokens. At the current frontier, raising the budget from 2.5 million to 50 million tokens increased the estimated horizon from roughly 2 hours to 14 hours.

Those figures do not mean every user will run 50 million-token agents casually. AISI did not publish a general dollar cost for those runs in the blog post, and cost will vary by model, pricing, routing and how much of the budget is actually consumed. The policy point is still clear: as inference gets cheaper, high-budget behavior becomes more relevant to real deployment.

A government lab built to see pre-release systems

AISI is unusual in this story because it is not a benchmark startup trying to sell the industry a leaderboard. It is a public-sector research institute with a security mandate. AISI says it tests leading AI systems before and after release, collaborates with AI companies on safety and security, and informs policymakers in the UK and allied governments.

The institute has also been deliberately staffed like a technical lab rather than a conventional policy unit. AISI says it has more than 100 technical staff, 66 million pounds in funding per financial year, priority access to more than 1.5 billion pounds of UK compute resources, pre-deployment access to leading AI models and more than 15 million pounds in grant capacity. Its leadership includes interim director Adam Beaumont, formerly GCHQ's chief AI officer; CTO Jade Leung, previously at OpenAI; chief scientist Geoffrey Irving, who has worked at OpenAI and Google DeepMind; and Hogarth, the Songkick co-founder and investor appointed in 2023 to chair the UK's AI Foundation Model Taskforce.

That institutional position explains why AISI's measurement work has consequences beyond academic benchmarking. If governments are going to use evaluations to make release, procurement or national-security judgments, the budget behind a score becomes part of the evidence. A pass rate without a compute curve is an incomplete risk signal.

The private market should read it the same way. Agent startups routinely claim they can automate engineering, security, research or back-office workflows. If their proof is a benchmark score at a fixed token cap, the missing question is whether the product improves smoothly when allowed to spend more time and tokens, or whether it plateaus quickly. Those are different businesses. One scales with inference economics. The other depends on model progress or product redesign.

The next benchmark disclosure

AISI's recommendation is practical: report capability curves, specify protocol choices, evaluate across multiple budgets and distinguish a low-capability model from an under-resourced run. The institute says it is already using multiple budgets, including large budgets for hard tasks, and working on minimum informative budgets and methods to forecast expensive high-budget performance from cheaper runs.

The unresolved issue is cost. A 10 million or 50 million-token run may be technically informative while still uneconomic for many products. Benchmarks that add compute curves should report price or resource consumption alongside performance, otherwise the market will replace one incomplete number with another.

Still, the measurement direction is hard to avoid. Agents are built to use tools, iterate and recover. A benchmark that cuts them off at a single arbitrary limit is measuring the cap as much as the agent. AISI's work gives evaluators a better habit: show the whole curve, then let buyers, labs and policymakers decide what part of it is affordable, deployable and safe.