Why Are Researchers Slamming OpenAI’s GPT-4 Paper?

The 98 page paper introducing GPT-4 proudly declares that they're disclosing ‘nothing’ about the contents of their training set
Listen to this story

After much speculation, OpenAI has finally announced GPT-4, potentially the next breakthrough in AI, after ChatGPT. According to OpenAI, GPT-4 is more reliable, creative, and able to handle much more nuanced instructions than GPT-3.5, the Large Language Model (LLM) behind ChatGPT.

However, researchers are not happy with the technical paper on GPT-4 released by OpenAI. On the very second page of the lengthy report, OpenAI claims, “ this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.”

Why Are Researchers Unhappy?

AIM Daily XO

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Many researchers and data scientists are left frustrated after reading through the almost 100 pages long GPT-4 paper, which fails to provide the bare minimum of information. OpenAI has also not mentioned the parameter count for GPT-4.

“In scientific papers it is generally expected that these details are mentioned because the papers are supposed to be reproducible,” Data Scientist Dr. Valeriy Manokhin, told AIM.

Download our Mobile App

AI researcher Sebastian Raschka, while sharing his disappointment, said, “We don’t learn anything about the model except “GPT-4 is a Transformer-style model pre-trained to predict the next token in a document, using both publicly available data (such as internet data) and data licensed from third-party providers.”

Mark Tenenholtz, data scientist at Stealth, said that the GPT-4 paper is highly disappointing since there are no discussions about the method, training stabilisation etc.

(Source: Twitter)

Another AI researcher points out, “What knowledge do researchers even gain from this? Nothing. It’s super frustrating.”

Emily M. Bender, Professor of Linguistics at the University of Washington, also points out that there is no mention of GPT-4 carbon footprint as well.

Is it even a scientific paper?

When a scientific paper gets published, details such as architecture, hardware, training compute, dataset construction, training method etc are always mentioned. It’s an important step because besides reproducibility, it also leads to fair comparison, encourages competition and drives innovation. 

Ben Schmidt, VP of Information Design at Nomic AI, says the paper is against the very principles of OpenAI. “The 98 page paper introducing GPT-4 proudly declares that they’re disclosing ‘nothing’ about the contents of their training set.”

The selection of training data can perpetuate historical biases and cause various forms of harm. Schmidt believes, in order to mitigate such harm and make informed judgments about where a model should not be applied, it is crucial to understand the types of biases embedded within the data. However, with OpenAI not disclosing such information, it is now possible.

AI critic Gary Marcus, in his newsletter calls recent developments a step backwards for science. “It sets a new precedent for pretending to be scientific while revealing absolutely nothing,” he said.

“Yesterday, AI became about corporate self interests. A divorce from the broad AI research field that made these companies even possible,” William Falcon, CEO at LightingAI, tweeted.

Pedro Domingos, Professor at University of Washington points out that the GPT-4 does not even mention the names of the authors, instead, “It has a long credits roll at the end, like a Hollywood movie,” he said. 

OpenAI also tested GPT-4 on some of the top professional and academic exams in the US designed for humans. OpenAI claims GPT-4 exhibits human-level performance on the majority of these exams.

( Source: GPT-4 paper)

However, Bender believes, “They seem to think this is a point of pride, but it’s actually a scientific failure. No one has established the construct validity of these exams vis-a-vis language models.”

A Corporate Approach

Overtime, OpenAI is behaving more like a corporation, and deviating more and more from the core principles, with which the organisation was founded in 2015. 

Elon Musk, who invested in OpenAI in 2015, also pointed out the same thing. “I’m still confused as to how a non-profit to which I donated USD 100 million somehow became a USD 30 billion market cap for-profit,” he tweeted.

OpenAI’s partnership with Microsoft also solidifies the claims that OpenAI is becoming more and more product oriented. Yannic Kilcher, CTO at DeepJudge claims that the GPT-4 paper mentions that the model is safe, but not safe for humanity or against biases, but safe to be used as a product.

In fact, OpenAI, in the paper, has claimed the competitive landscape and the safety implications of large-scale models like GPT-4 as reasons for withholding information about the model.

Sign up for The Deep Learning Podcast

by Vijayalakshmi Anandan

The Deep Learning Curve is a technology-based podcast hosted by Vijayalakshmi Anandan - Video Presenter and Podcaster at Analytics India Magazine. This podcast is the narrator's journey of curiosity and discovery in the world of technology.

Pritam Bordoloi
I have a keen interest in creative writing and artificial intelligence. As a journalist, I deep dive into the world of technology and analyse how it’s restructuring business models and reshaping society.

Our Upcoming Events

24th Mar, 2023 | Webinar
Women-in-Tech: Are you ready for the Techade

27-28th Apr, 2023 I Bangalore
Data Engineering Summit (DES) 2023

23 Jun, 2023 | Bangalore
MachineCon India 2023 [AI100 Awards]

21 Jul, 2023 | New York
MachineCon USA 2023 [AI100 Awards]

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

Council Post: The Rise of Generative AI and Living Content

In this era of content, the use of technology, such as AI and data analytics, is becoming increasingly important as it can help content creators personalise their content, improve its quality, and reach their target audience with greater efficacy. AI writing has arrived and is here to stay. Once we overcome the initial need to cling to our conventional methods, we can begin to be more receptive to the tremendous opportunities that these technologies present.