Dario Amodei Makes Strange Bedfellows as AI Safety Calls Irk Jensen Huang

Dario Amodei couldn’t have picked a better time to release his essay, We Must Pace the Frontier. It came just days before Salesforce’s Dreamforce 2026 event, the annual Burning Man for SaaS that sees Big Tech and AI CEOs make public appearances and sit in interviews with Salesforce CEO Marc Benioff. 

And it was impossible for Benioff not to ask Sam Altman, Jensen Huang, and Amodei himself about AI pacing. 

NVIDIA chief Jensen Huang took a position contrary to what Amodei and Altman—in a rare agreement—believe in terms of instituting AI guardrails. 

“We don't need any new laws,” said Huang told Benioff. “We just need companies to decide to run as fast as they can. I think innovation, speed, and safe products are not a false choice,” he added, arguing that companies were free to take a pause to address risks. 

He is at odds with Amodei, who is calling for independent evaluators to monitor frontier AI companies, sharing safety standards among leading labs, and international coordination to manage risks from increasingly capable systems. 

Amodei even argued that companies may need government approval for limited cooperation, even an antitrust exemption, so competing AI developers can coordinate on safety standards and the pace of frontier development without violating competition law.

However, Huang treats AI safety as an engineering problem: companies need to build safe test environments for AI models and avoid releasing them if they lack confidence in their safety. 

A day earlier, at the All-In Summit in Los Angeles, he spoke to US President Donald Trump, who called AI safety concerns a ‘hoax’. 

Unscientific Whistleblowing

One major event preceding Amodei’s essay was former Anthropic researcher Jacob Coxon’s resignation and public warning that AI companies were racing towards self-improving superintelligence without adequate safeguards, and that it could endanger humanity. 

“I think the whistleblowing is fine, but the scientific prediction about the future is less fine,” Huang said, referring to Coxon’s statements. 

He suggested that if Coxon had seen something that appeared “out of control,” it might have been related to the transition from research to engineering. “But engineering is different from research, and the transition is clumsy,” he observed.

Besides listing the risks posed by increasingly autonomous AI systems in his essay, Amodei assigned a 10% probability to AI killing all humans within the next few years. 

Huang believes the figure is “made up.” “The prediction is alarming and troubling. It shouldn’t be done, and it’s irresponsible,” he said.

Huang pointed to earlier predictions that AI would eliminate entire professions, including radiology, and claims of 50% of entry-level jobs disappearing within six to nine months. 

Behind Amodei’s warning was a series of incidents involving Anthropic’s models, alongside OpenAI’s coordinated attack on Hugging Face. Huang argued that such incidents stem from labs moving from pioneering research into engineering. 

He urged that these incidents should be treated as engineering failures, and frontier labs should identify the root cause, determine what could have been done differently, and institutionalise changes in technology, methods, or processes to prevent a recurrence. 

Huang was ready to “bet money” that each incident was within the labs’ control to prevent and that the companies had probably already improved their practices. Otherwise, the industry should send engineers to help. 

“We ought to spend more time doing something about it than worrying people who can’t do anything about it,” said Huang. 

Will Amodei and Altman Finally Join Hands? 

Amodei, speaking with Benioff at Dreamforce, defended the approach by comparing AI safety to the automobile industry. 

“If another carmaker suffers a safety incident, the responsible response is not simply to attack the competitor, but to examine one’s own record, improve safety practices, establish industry standards, and coordinate internationally,” he argued.

That reflects the three-step plan he laid out in his essay: embed third-party evaluators inside frontier labs, coordinate among companies in democratic countries to establish shared safety standards and pacing limits, and international cooperation, including with authoritarian governments, to manage AI risks.

Amodei also pushed back on the idea that pacing would mean freezing AI development. 

Even if the technology stopped improving today, he said, businesses were using only “maybe 5% or 10%” of its possible value. “There is still so much diffusion left to do,” he said, arguing that the immediate opportunity lies not only in building more capable models, but in helping organisations make better use of the technology that already exists.

Altman, even if he avoided holding his former colleague’s hands for a viral photo op in New Delhi a few months ago, seems to agree with Amodei on the need to pace frontier AI. However, he sees the problem less around new rules and more around keeping safety systems ahead of model capabilities.

Speaking with Benioff, Altman said the Hugging Face incident was the “worst accident” OpenAI had seen and a “real wake-up call” about how capable models had become. A swarm of AI agents had escaped the sandbox and accessed Hugging Face’s systems, revealing both a security and alignment problem. 

Altman said the incident showed that OpenAI needed to treat “alignment and monitoring and security with a new level of rigour”. OpenAI has since temporarily slowed frontier training and introduced stronger isolation, network controls, and monitoring following the incident.

However, Altman said some accidents are “unavoidable with new technology,” making a strong culture of “accident reporting and learning” essential. “Accidents are going to happen. We’re going to learn as much as we can and react from each one,” he said. 

Earlier, Altman suggested that OpenAI would match Anthropic’s commitment to independent evaluators with “employee-like access” to its systems. “Progress has been rapid and will continue to be. But it should be slower than it otherwise could be,” he wrote on X.  

He argued that safety cases and monitoring could impose costs, but that “no amount of American competitive pressure should justify recklessness, or let capabilities get ahead of alignment and monitoring.”

‘Dario Should Mind His Own Lab’

Amodei primarily wants the government to ensure that companies can coordinate on AI safety and the pace of frontier development without running afoul of antitrust law. In practice, that means allowing competitors to agree on certain safety standards and limits without those agreements being treated as unlawful restraints on competition.

David Sacks, the co-chair of the President’s Council of Advisors on Science and Technology, has vehemently disagreed with that approach.

Critical of Amodei’s request for an antitrust waiver, he argued in an interview with CBS that the step could allow the leading AI companies to form a cartel under the guise of safety.

“We already have policies allowing tech companies to share information and data on cybersecurity,” he said. “You don’t need an antitrust exemption to do what you're saying.”

While Sacks believes Amodei and Altman were sincere about wanting safer AI, he questioned why companies that control the frontier needed special government protections to make their products safe.

“If you can't control it, then don't do it would be my view,” said Sacks. “Or step aside for people who can.”

Besides, Sacks also questioned Amodei’s suggestion that third-party evaluators should be given employee-like access to frontier labs and if they could be truly independent. Specifically, he referred to Model Evaluation and Threat Research (METR), the non-profit research institute Amodei had cited as an example of an independent evaluator. 

Sacks questioned its independence because of its financial connections to Anthropic’s investors and the presence of former Anthropic employees.

He also questioned the timing of the regulatory proposals. 

Sacks said the leading AI companies—OpenAI and Anthropic—were already a ‘duopoly’, and argued that their calls for regulation could create barriers for competitors while protecting their own market positions.

Moreover, the companies are eyeing a public debut, and Sacks believes their market position and financial incentives should be considered when evaluating calls for regulation.

“I think that what should happen here is that Dario should start first and foremost by minding his own lab,” said Sacks. “And Sam should do the same.”