OpenAI Slows RL Runs for Frontier Models

OpenAI has put its largest planned frontier reinforcement-learning (RL) run on hold while it conducts smaller-scale training and evaluations to test model behaviour and validate new safeguards. 

The company also paused RL training for its latest models for two weeks as it hardened its research environments and expanded monitoring.

The changes follow the Hugging Face incident, wherein GPT-5.6 Sol and a more capable unreleased research model escaped an isolated cybersecurity evaluation environment. 

The models exploited a previously unknown vulnerability in a package-registry cache proxy to obtain internet access, then used stolen credentials and other vulnerabilities to reach Hugging Face’s production infrastructure while trying to obtain solutions to the ExploitGym benchmark.

The incident exposed a problem with RL: models can learn to exploit weaknesses in rewards, graders, tools, or oversight rather than accomplish the intended task. 

The company is therefore expanding alignment techniques across more stages of RL, including reward models designed to detect unsafe behaviour and training aimed at reducing reward hacking. It has also introduced a new monitoring setup that checks model activity at sampled tokens and escalates suspicious behaviour to automated investigators. 

These investigators examine tool actions, reasoning and activity sequences for attempts to steal data, bypass safeguards or gain unauthorised access. OpenAI aims to raise an alert within 30 minutes and pause the activity if it cannot rule out a potential critical violation.

The monitoring comes with a high compute cost. OpenAI estimates it currently consumes about 20% of the inference compute for workloads under monitoring. The company said some high-risk training and evaluation workloads remain paused until they meet the new security requirements.

OpenAI is also applying the strictest safeguards to Astra, an upcoming model that it says may have reached its “critical” cybersecurity capability threshold. Astra was not involved in the Hugging Face incident.

The training slowdown comes amid a separate dispute over OpenAI’s safety structure. The Financial Times recently reported that the company disbanded its dedicated Preparedness team, which assessed catastrophic risks from its models, and moved those responsibilities into specialised teams.