OpenAI Unveils CriticGPT to Review GPT-4’s Performance

In taking a different approach to assessing GPT-4’s performance, OpenAI has announced the creation of CriticGPT.

Based on GPT-4, the model is able to assess and critique ChatGPT’s responses, assisting human supervisors and trainers in improving the model. “We found that when people get help from CriticGPT to review ChatGPT code, they outperform those without help 60% of the time,” OpenAI stated.

Currently, CriticGPT is in the process of being integrated into ChatGPT’s Reinforcement Learning from Human Feedback (RLHF) labelling pipeline. With the added assistance from the new AI critic, OpenAI is hoping to improve the efficiency of detecting and rating issues within training data and responses produced by ChatGPT.

The model’s creation stems from a research paper released by OpenAI, with one of the researchers involved being former OpenAI safety head Jan Leike, who left the company this year in May along with Ilya Sutskever to build Safe Superintelligence Inc. The paper, titled ‘LLM Critics Help Catch LLM Bugs,’ found that critic models were able to identify flaws and errors in ChatGPTs training data that were otherwise labelled “flawless”. 

Interestingly, in 2022, Leike, under OpenAI, released a similar research paper on building self-critiquing models to assist human evaluators.

Concerning the need to develop such a model, OpenAI stated that as AI behaviours become more advanced, mistakes within these models become more subtle. “This can make it hard for AI trainers to spot inaccuracies when they do occur, making the comparison task that powers RLHF much harder.

“This is a fundamental limitation of RLHF, and it may make it increasingly difficult to align models as they gradually become more knowledgeable than any person that could provide feedback,” the company stated.

The model was trained similarly to ChatGPT, based on RLHF, however, it was exposed to exponentially more inputs that include mistakes which it was tasked with critiquing. Regarding its effectiveness, OpenAI stated that while it was not always accurate, the introduction of the model into the RLHF process allowed trainers to better detect issues with model-written answers.

In terms of limitations, the company has said that work is currently being done to develop models that will be able to understand and critique longer and more complex tasks, as well as reduce hallucinations that are still present in CriticGPT.