There has been a mysterious chatbot going around on the internet, with powerful capabilities, ones that people say can put GPT-4 to the test. im-good-gpt-2-chatbot was on the LMSYS Org, a benchmarking website for testing models. It abruptly disappeared last week, but is now back again on the website.

To fuel everyone’s curiosity, OpenAI chief Sam Altman posted on X, “I do have a soft spot for gpt2”.

After this cryptic post, Altman posted again, “im-good-gpt-2-chatbot”, leaving everyone wondering if the model was actually created by OpenAI, and perhaps that the company was just testing out the next version of its LLM in the open. 

In light of these speculations, Altman replied to another post which said “i’m a bad gpt4 chatbot” with “you-are-not-a-good-user”. This is the same response that Bing Chat used to give when being tested and was internally code named Sydney.

All this points towards the ‘mysterious gpt-2-chatbot’ being indeed made by OpenAI or Microsoft. 

How good is good-gpt-2?

Everyone is testing out the model capabilities. Min Choi was able to create a Flappy Bird clone with just a single prompt and three images. 

Some say that it is a smaller version of GPT-5, while others say that it is just another model. It is not even clear yet if it is actually created by OpenAI, which is highly unlikely. Making things interesting, a user on Reddit posted a screenshot claiming that the model retrieved information from the OpenAI website. 

It is also quite possible that OpenAI upgraded GPT-2, the 1.5 billion parameter model with sophisticated fine-tuning on synthetic data by newer models. Moreover, it also uses OpenAI’s tiktoken tokenizer and also similar prompt injection vulnerabilities when compared to others such as Mistral or Meta.

According to several developers, the model also works very similar to the OG GPT-2 by OpenAI with increased reasoning and multimodal capabilities. In an experiment, the model was able to solve a freshman physics problem that all other models, including GPT-4 Turbo, were unable to solve.

But when it comes to more such capabilities, the same screenshot also showcases 250,000 tokens per minute, which is surprisingly slow when compared to other open source models in the arena, or when it comes to OpenAI’s expertise.

Moreover, some experts say that if this is indeed a smaller or teaser version of GPT-5, it would hugely impact OpenAI’s AI superiority as the onset of Llama 3’s next version would overshadow it. Sully on X posted that the model is barely better than GPT-4 as per his evaluations. 

Release GPT-5 already

Altman had recently said that in the coming months, GPT-4 would be the worst model when compared to what the company is building. If true, it means that all the other models trying to compete with GPT-4 are much worse than what OpenAI has been building.

In a recent talk at Harvard University, Altman said that the secret chatbot is not GPT-4.5, which a lot of people were predicting. Some even predicted that it could be a version of Microsoft’s Phi-3 model, which was released just a few days ago. 

According to several speculations, OpenAI is also expected to soon release its search features on ChatGPT, giving it close competition to Google and the likes of Perplexity. The possibility of gpt-2-chatbot, a small and lightweight model to be tested out with that is also not completely undeniable. Moreover, the company also might be testing the model for its upcoming Apple partnership for on-edge use cases. 

Adding to this, there has been a lot of hype around the new AI models by other companies such as Meta, Databricks, and Anthropic, which may have forced OpenAI to release the cryptic model out in the open to show that it is still on the lead of others.

It is high time that OpenAI releases GPT-5, given that Altman believes in releasing models in a staggered fashion, instead of all at once.