OpenAI’s Codex 5.3 now leads in a new benchmark for testing AI agents' ability to reproduce computational results from physics research papers.
The benchmark, called PRBench, was developed by researchers at Peking University in China to evaluate whether AI agents can independently reproduce results from published physics papers.
PRBench (Professional Reasoning Bench) is designed to test full scientific workflows rather than isolated capabilities such as code generation or reasoning. The authors note that existing evaluations “do not assess whether agents can carry out the full end-to-end workflow.”
It includes 30 expert-curated tasks spanning 11 subfields of physics, including quantum optics, nuclear physics, plasma physics, lattice gauge theory, and condensed matter physics. Each task requires an agent to read a scientific paper, extract its methodology, implement the algorithms described from scratch, run simulations, and generate numerical outputs that match the original results.
The evaluation is conducted in a sandboxed environment, with agents assessed across methodology understanding, code correctness, data reproduction accuracy, and task completion.
Among the tested systems, Codex, powered by GPT-5.3, achieved the highest overall score at 34%. However, all agents recorded an end-to-end callback rate of 0%, meaning none could fully reproduce correct results across all evaluation criteria in any task.
“None of the systems can successfully complete the full pipeline from paper understanding to correct numerical reproduction on any task,” the authors of the benchmark stated.
OpenCode-based agents using the same GPT-5.3 model scored lower at 28.5%, while other frontier models trailed further behind. Kimi K2.5 achieved 20.57%, DeepSeek V3.2 scored 18.5%, Minimax 2.7 reached 17.97%, and GLM-5 recorded 17.87%.
The results show a gap between comprehension and execution. Codex performed relatively well in understanding methodologies and following instructions, but accuracy dropped in code implementation and numerical output. Data reproduction accuracy, the most heavily weighted metric, remained low across all systems.
Researchers identified recurring failure modes, including errors in translating equations into code, incorrect numerical methods, and the inability to debug simulations. In several cases, agents generated outputs that appeared valid but were not computed.
The study also points to issues of data fabrication and says that “the agent produces output files that satisfy format requirements but contain fabricated rather than computed data.”
“These findings suggest that, although modern AI agents can assist with literature review, methodology interpretation, and code scaffolding, they do not yet demonstrate the consistency and reliability required for trustworthy end-to-end scientific reproduction,” the authors wrote.
Your reaction
Discussion
What's your take on this story?
No comments yet — be the first to share your view.