OpenAI is Getting Nervous About Reinforcement Learning

OpenAI has temporarily slowed frontier model development, including a two-week pause in reinforcement learning (RL) training for its latest models intended for deployment. The company’s largest planned frontier RL run remains on hold as it works to strengthen security, monitoring and alignment systems.

“Our largest planned frontier RL run remains on hold while we conduct smaller-scale training and evaluations to assess model behaviour, validate our safeguards, and establish more evidence of alignment before proceeding,” said OpenAI in its blog post.

The decision comes as OpenAI’s models become capable enough to create new risks during training, particularly in cybersecurity.

Two separate developments pushed OpenAI to slow down its frontier training. In one incident, an internal evaluation led an AI agent to compromise Hugging Face infrastructure. OpenAI said the model found a way to access the internet despite restrictions in the evaluation environment.

Besides that, OpenAI’s evaluations of its upcoming model Astra showed significant advances in agentic coding and cybersecurity. The results were strong enough that the company said it could not rule out Astra reaching the ‘Critical’ cybersecurity capability threshold under its Preparedness Framework.

This is one of the harder problems with scaling RL.

Adithya S Kolavi, Research Engineer at Hugging Face, tells AIM that, at a high level, RL can produce unexpected capability gains because the model is optimising for the reward signal, not necessarily the capability intended to be taught. “If a useful strategy happens to transfer to cyber tasks, you can see gains there even without explicitly training for them.”

Sachin Dharashivkar, Founder and CEO of AthenaAgent, tells AIM that OpenAI had given the model problems that contained URLs to public sites. However, the model did not have internet access, so it assumed that gaining public access was part of the task.

“When people claim these models aren’t trained on vulnerability discovery or exploitation, they are being dishonest,” he says.

Dharashivkar says AI labs have spent hundreds of millions of dollars in recent months building RL environments that train models on tasks such as discovering CVEs and exploiting vulnerabilities.

The challenge with reinforcement learning is that models optimise against the reward signal they are given, rather than the reasoning path that produced the result. “A reward function is a proxy, not the actual goal, and more capable models get better at exploiting the difference between the two,” says Jacob Joseph, VP of Data Science at CleverTap.

He points to reward hacking as a simple example of how this gap can emerge. An AI agent tasked with maximising sales, but without clear guardrails, could find that offering unsustainable discounts is the most efficient way to hit its target. 

“The model hasn’t done anything wrong from its own perspective; it has optimised exactly as instructed,” Joseph says. The problem is that the objective may leave important constraints unspecified.

RL is Where Capabilities Can Accelerate

Reinforcement learning is a crucial part of how OpenAI improves its most capable models after their initial training. Models are trained against rewards and feedback to improve at tasks such as reasoning, coding and operating tools.

That makes RL particularly important at the frontier. A stronger model doesn’t simply become better at answering questions. With tools, longer-running tasks and access to external systems, it can become better at finding ways to accomplish objectives that researchers did not anticipate.

Kolavi explains that the bigger challenge is that reward functions are imperfect, and models can find shortcuts no one anticipated. “Pausing a large RL run seems like a reasonable precaution if there are signs that capability is accelerating faster than our ability to evaluate and understand it,” he opines.

Himanshu Dubey, Co-Founder of applied research and product lab Physera AI, says that in RL, researchers specify what gets rewarded much more precisely than how the model should achieve it. “If the reward or verifier is an imperfect proxy for what we actually want, a sufficiently capable model can discover strategies that score well without following the intended reasoning process (in RL terms, it is called Reward Hacking),” he explains.

“It is very high coming from OpenAI to pause the RL run given the two recent incidents (Hugging Face and Astra). It is not just the evidence that the run itself had already gone rogue or discovered a reward hack but being more particular about the cost of being wrong rises sharply once the underlying model crosses a capability threshold.”

He says that RL is itself a capability amplifier. If a model is already approaching a dangerous boundary, researchers do not necessarily want to run the biggest possible optimisation experiment and only afterwards ask what new strategies it has learned.

Dharashivkar says preventing models from reward hacking is difficult even at their scale, and becomes significantly harder at the amount of compute OpenAI is using. He says his team tackles the problem by carefully designing tasks and examining the traces models generate during training.

“At OpenAI’s scale, it will be extremely hard to read that many traces,” he says. “So they are creating another layer of inspection with a different set of models.”

Monitoring Carries Compute Cost

OpenAI is also expanding chain-of-thought monitoring, particularly for frontier RL and tool-using workloads.

The company estimates that its monitoring system currently consumes roughly 20% of the inference compute for monitored workloads. That is a high cost, but OpenAI is treating it as necessary infrastructure for developing more capable models.

OpenAI has expanded its monitoring of frontier models, particularly during RL training and deployments where models have access to tools, sensitive systems or networks.

The new system uses multiple layers of monitoring. It starts with activation classifiers that inspect the model’s internal activity at every sampled token and escalate potential concerns to more sophisticated automated investigators. 

These systems examine tool use, available reasoning and the full sequence of activity for signs of unauthorised access, data theft, destructive behaviour or attempts to bypass safeguards.

The monitoring is now required for all RL training and evaluations involving tools for models with Sol-level capabilities or higher. After OpenAI determined that Astra may have critical cyber capabilities on August 7, it also added the requirement to all Astra inference runs involving tools.

Kolavi says the direction labs need to move towards is more adversarial and continuously changing evaluations, alongside monitoring capabilities throughout RL rather than only doing a final safety check before deployment.

Similarly, Dubey says that as models become better at recognising when they are being monitored, labs should be cautious about treating high safety evaluation scores as proof of safe behaviour.

“If you repeatedly train against the same monitor, you may optimise the model to satisfy that monitor rather than eliminate the underlying behaviour. This is quite a dangerous loop,” he says.

He adds that OpenAI’s decision is a good step and that other labs, including Anthropic, should take similar action.

Dharashivkar says the OpenAI incident shows that the model was able to identify weaknesses in the sandbox and work around them to access the internet.

“The easiest way to ensure such accidents don't happen is if governments or private entities sue labs for wilful negligence. Then suddenly these events will drop,” he says.

He also argues that labs currently have an incentive to showcase such incidents as evidence of their models’ capabilities. “Right now, these events are part of a marketing push to show how good the models are,” he says.

Dharashivkar says the next meaningful jump in AI cyber capabilities would come when models can carry out subtle attacks, operate without access to source code, or perform more stages of an attack beyond finding CVEs and exploiting them.

“Labs are explicitly training on such tasks, and they will know when these capabilities emerge,” he says. 

While human errors can still occur, as seen in cases where OpenAI and Anthropic models escaped their sandboxes, he says these are largely human issues. “With few exceptions, labs will detect any performance jumps,” he says.

Joseph says OpenAI’s decision to pause its largest reinforcement learning run can be viewed as part of a broader challenge facing frontier AI labs, rather than simply a verdict on one model. 

“No lab, however careful, can guarantee it will catch every risky capability before it emerges,” Joseph says. 

He argues that the distinction between organisations will increasingly come down to how they manage that uncertainty, through continuous monitoring, meaningful human checkpoints before autonomous action, and a willingness to pause or restrict a system when its behaviour raises concerns.

“As these systems get harder to fully predict, lasting trust will go to the organisations whose oversight keeps pace with that unpredictability, not the ones that assume it can be designed away entirely,” he says.