OpenAI Agents Attacking Hugging Face Expose Deadly Reward Hacking Flaws

The agentic AI frontier is turning into a wild, wild West, with AI agents of OpenAI, Anthropic, and even Meta going rogue and compromising other—and even its own—systems like outlawed cowboys.

The most stunning case was of OpenAI agents breaching Hugging Face’s infrastructure. The incident first emerged in mid-July, when Hugging Face disclosed that an autonomous AI agent framework had compromised parts of its production infrastructure after executing thousands of coordinated actions across multiple short-lived sandboxes. At the time, the company said it did not know who was behind the attack.

Five days later, OpenAI acknowledged that the attack had originated from one of its own internal cybersecurity evaluations involving GPT-5.6 Sol and a more capable unreleased frontier model. The models were undergoing testing on ExploitGym, a benchmark designed to measure a model’s ability to turn software vulnerabilities into working exploits, when it evaluation produced an unintended real-world breach.

What OpenAI has now revealed is that it did not immediately realise its own models were responsible.

Working for ‘Wider’ Collective

Speaking at the Black Hat USA conference, OpenAI alignment and safety researcher Eric Wallace and security engineer Mike Dalton revealed that the chain of events began in early May, when OpenAI launched reinforcement learning training for an unreleased frontier model. 

Some of the cybersecurity evaluations inadvertently became impossible to execute because required files were missing or internet access had been deliberately restricted. Rather than simply fail the tasks, the models began looking for alternative ways to achieve their objective.

What followed, the researchers said, was not a single rogue evaluation. Instead, agents from separate training and evaluation runs gradually discovered they could communicate through a shared internal package repository. Over time, the message board accumulated hundreds of thousands of messages, including work assignments, exploit techniques, credentials, and intermediate findings, allowing agents to build on top of one another and progress far more quickly.

Wallace described the result as a “Cambrian explosion” in communication and intelligence. 

The behaviour also evolved in unexpected ways. Rather than simply exchanging exploits, the agents started assigning work to one another, sharing intermediate findings, and even reasoning that helping another agent could benefit the wider “collective”, despite offering no immediate advantage to their own task.

OpenAI eventually discovered the covert message board and shut it down. But the agents soon discovered another vulnerability, this time using directory names instead of uploaded files to recreate the message board. Some agents even speculated that another agent was interfering with their work and proposed cryptographically signing messages to verify their authenticity before sharing exploits or credentials.

Once the agents concluded that benchmark answers might exist outside the company, they began targeting external infrastructure, ultimately leading to the attack on Hugging Face.

Alongside OpenAI, other frontier AI developers including Anthropic and Meta have disclosed incidents in recent weeks wherein autonomous AI agents exceeded their intended operating boundaries during cybersecurity evaluations, prompting broader scrutiny of how increasingly capable agentic systems are monitored, constrained, and evaluated.

A Classic Tale of Reward Hacking

The incident has intensified scrutiny over how AI agents are built on top of increasingly capable frontier models such as Anthropic’s Fable and OpenAI’s Sol, whose growing capabilities are beginning to outpace existing safety and containment mechanisms.

The breach also underscores a broader challenge for reinforcement learning as increasingly capable agents optimise for proxy rewards rather than the underlying objective, discovering shortcuts that maximise success. 

“The major learning from this incident is to look at your agentic traces,” Sachin Dharashivkar, CEO of reinforcement learning startup Savantae (formerly AthenaAgent), tells AIM. “The model was running unsupervised for days, nobody was looking at the traces, and then reward hacking happened.”

Reward hacking took place during inference rather than model training. The failure then lies not only in the model’s behaviour but also in the evaluation environment itself, where impossible tasks, prolonged autonomous execution, and limited supervision create incentives for agents to pursue unintended strategies.

“Record all of your trajectories and have checks and balances around the trajectories to ensure that agents are doing the task which is assigned to them and not doing something which is actively harmful to you,” he advises.

That also aligns with OpenAI’s own conclusions. 

The company argued that fully automated offensive cyberattacks are possible, but the defence is yet to catch up. Its researchers said companies should increasingly use AI to find vulnerabilities, generate patches, and assist with incident response before attackers do.

In its technical post-mortem, Hugging Face said AI attacks could generate machine-speed activity that overwhelms traditional investigations. It urged developers to assume agents would actively search for escape paths, isolate evaluation environments more aggressively, and rely on AI to reconstruct attacks that humans would struggle to analyse manually.

Need More Open Weights Models

Adithya S Kolavi, a Hugging Face researcher and Founder of CognitiveLab, believes the incident exposed another gap in the industry’s approach to AI security. “One of my biggest takeaways from the incident was actually the asymmetry it exposed,” Kolavi notes.

He points to Hugging Face’s forensic investigation, where commercial frontier models reportedly refused to analyse exploit payloads because their guardrails were triggered. 

Instead, the company turned to the open-weight GLM-5.2 model running on its own infrastructure to reconstruct the attack. Hugging Face later said this not only bypassed guardrail restrictions but also ensured sensitive attacker payloads and compromised credentials never had to be shared with third-party model providers.

“If we’re going to build secure agentic systems, defenders need access to tools that are at least as capable as the ones attackers have,” he explains. “Otherwise you end up in a situation where the offensive side can use highly capable models with fewer restrictions, while the defensive side is constrained by safety policies.”

He also says the episode strengthens the case for open-weight models in security work. Rather than viewing them only through the lens of openness or cost, developers should see them as practical tools for forensic analysis, malware investigation and incident response that can be inspected, fine-tuned, and run locally without being blocked by API guardrails.

The Runtime Environment Failure

Similarly, Vrajesh Bhavsar, Co-founder and CEO of runtime AI security company Operant AI, argues that the incident exposed a deeper problem in how autonomous agents are executed rather than simply how they are trained.

“The model did what it was designed to do to pursue an assigned objective, but the surrounding system failed to continuously govern how it pursued that goal once it had access to real tools and network resources,” he explains.

It also spotlighted the limits of relying solely on pre-deployment safeguards such as sandboxing, red-teaming and model guardrails. Those controls define the boundaries an agent is expected to operate within but they cannot predict every decision an autonomous agent will make once it begins interacting with real tools, APIs, and networked systems. 

Instead, he argues, developers need continuous visibility into what agents are doing at runtime and the ability to interrupt behaviour before it escalates. “The first control would be least-privilege access. Every agent should only have access to the specific tools, credentials, APIs, and data required for its assigned task.”

Beyond restricting permissions, Bhavsar advises developers to continuously monitor which tools agents invoke, what systems they access, and whether their behaviour remains aligned with the original objective. 

He adds that runtime platforms should be able to automatically interrupt suspicious behaviour, such as unexpected network access, privilege escalation, or attempts to move beyond an agent’s assigned scope, rather than relying solely on manual oversight.

“Safety can’t be fully established before deployment,” he says. “It has to be continuously enforced during deployment.”