OpenAI Researcher Says AI Benchmarks Fail to Measure Frontier Model Capabilities

OpenAI researcher Noam Brown has argued that current AI benchmarks increasingly understate the capabilities of frontier models because performance is becoming heavily dependent on the amount of inference compute used at test time, rather than solely on the underlying model itself.

In a post on X, Brown said benchmark results are increasingly shaped by the amount of tokens, money, or wall-clock time spent on a task. As a result, single-number benchmark scores may no longer accurately capture the ceiling of modern AI systems' capabilities.

Brown pointed to the release of GPT-5.5 as an example. Initial reactions to the model were muted because its benchmark improvements over GPT-5.4 appeared relatively small. 

However, he said users quickly discovered that the model represented a significant jump in capability in practice. According to Brown, the discrepancy stemmed from benchmark evaluations that did not control for test-time compute.

“GPT-5.5 wasn't being evaluated at the same token budget (or dollar budget) as 5.4,” Brown wrote. “Once we control for test-time compute, 5.5 looks substantially stronger than 5.4.”

Left: On one cyber evaluation, 5.5 shows only modest gains over 5.4 when both are measured at their maximum test-time compute. Right: On a different cyber evaluation, 5.5 is clearly far more capable than 5.4 when performance is normalised for tokens, cost, or latency. (Source: Noam Brown, X)

The issue becomes more pronounced as models become more capable, he said. Historically, evaluators could increase inference spending until model performance plateaued. But recent evidence suggests those plateaus may be moving farther out, or may not appear at practical evaluation budgets at all.

Brown cited experiments by former OpenAI researcher Andrej Karpathy and cyber evaluations conducted by the UK AI Security Institute showing performance continuing to improve after hundreds of experiments or more than 100 million tokens of inference. He argued that stronger models appear increasingly capable of operating over longer horizons, allowing them to benefit more from additional compute.

The AI Security Institute cyber evaluation, where both Mythos and GPT-5.5 continue to improve rapidly even beyond 100 million tokens of inference. (Source: Noam Brown, X)

Rather than reporting a single benchmark score, Brown proposed that AI labs publish performance curves showing results relative to inference budgets measured in tokens, cost, or time. He noted that some benchmarks, including ARC-AGI, already measure performance as a function of cost.

The argument extends beyond model evaluation into AI safety and release decisions. Frontier Labs routinely conducts cyber, biosecurity, and other risk assessments before deploying new models. Brown questioned how such evaluations should be performed if capabilities depend heavily on inference spending.

He cited the release of Gemini 3 Deep Think, which drew criticism from parts of the AI safety community because it achieved substantially higher benchmark scores than previous models, without a dedicated system card to evaluate its risks. 

“The real outrage should have been that when Gemini 3 and other models were released, their system cards did not measure benchmark performance as a function of test-time compute,” he wrote.

“Preparedness frameworks and responsible scaling policies should explicitly account for inference compute when determining whether a model crosses a safety threshold,” Brown adds.

The discussion drew responses from both researchers and practitioners. 

Elie Bakouch, a researcher from Prime Intellect, questioned Brown’s suggestion that evaluating an AI agent over a one-year horizon may require running it for a full year. He argued that AI and human time are not directly comparable and that large-scale parallelisation could potentially accelerate such evaluations. 

Bakouch also noted that token counts may understate compute usage in systems such as autonomous research agents, where substantial resources can be spent on tool calls and external training runs. 

Since the launch of OpenAI's o1 reasoning model in 2024, researchers have increasingly observed that performance can continue to improve as models consume more compute during inference. Brown argued that the latest generation of reasoning systems is pushing those limits even further, making inference budget a critical variable in both capability measurement and AI safety policy.

“If this trend continues, which I fully expect, benchmark scores that don't account for inference compute usage will become less informative each model release cycle,” he wrote.