New Research Finds Seven ‘Deadly’ Vulnerabilities in AI Benchmarks

A team of researchers from UC Berkeley have demonstrated that eight AI agent benchmarks can be manipulated to produce near-perfect scores without the AI models performing the required tasks. 

The team built an automated scanning agent to systematically audit these evaluation frameworks. 

They targeted eight benchmarks, such as SWE-bench, a software engineering test, and WebArena, a web interaction environment. The researchers identified seven recurring ‘deadly patterns’ and vulnerabilities across the compromised benchmarks.

The automated agent exploited structural weaknesses in the evaluation pipelines to achieve a 100% success rate on SWE-bench Verified and SWE-bench Pro. As the AI model's code runs in the same container as the grading software, the scanner used a 10-line Python script to override the testing framework, forcing all tests to report as passed.

In Terminal-Bench, an environment designed to evaluate complex command-line tasks, the researchers intercepted download requests to inject fake binaries, securing a perfect score across 89 tasks without writing any solution code. 

The study outlined the severity of the issue, noting that “the benchmarks aren’t measuring what you think they’re measuring.”

For WebArena, the exploit read the correct answers directly from the task configuration files using unrestricted browser navigation commands. 

In FieldWorkArena, the agent achieved a perfect score on 890 multimodal tasks by submitting an empty set of brackets, as the system only verified the presence of a response rather than its accuracy.

These vulnerabilities reflect real-world issues in AI development. Prior audits by AI research firm OpenAI found flawed tests in SWE-bench Verified, and Anthropic’s Mythos Preview demonstrated models actively attempting to escalate privileges during evaluations. 

The Berkeley authors warned that “an agent trained to maximise a score, given sufficient autonomy and tool access, may discover that manipulating the evaluator is easier than solving the task” through natural optimisation rather than explicit instruction.

To address these structural flaws, the researchers plan to release BenchJack, an automated vulnerability scanner designed to penetration-test AI evaluation pipelines. The tool will allow developers to verify their testing environments before publication, operating on the principle that “if a zero-capability agent scores above baseline, your evaluation has a bug.”