OpenAI is likely to release weights of its models in the coming months. Amid the Llama fever, OpenAI’s Andrej Karpathy, recently said that all of this is quite generic to just transformer language models. “If/when OpenAI was to release models as weights (which I can neither confirm nor deny!) then most of the code here would be very relevant. 

In other words, OpenAI is most likely to make GPT- 3.5 open source according to OpenAI’s Karpathy, a prominent figure in the field of deep learning. It has to be noted that the company has not made any official announcement about this. The conversation stems from a Twitter (now X) thread, when one of the users asked Karpathy as to why has been playing with Llama 2, instead of building Jarvis for OpenAI. 

This new development comes in the backdrop of the recent release of Baby Llama aka llama.c, where Karpathy has been exploring the concept of running large language models (LLMs) on a single computer as part of his recent experiments, inspired by the release of Meta’s Llama 2. 

Check out the GitHub repository here. 

Karpathy said llama2.c can now load and inference the Meta released models. He further gave an example of  inferencing the smallest 7B model at ~3 tokens/s on 96 OMP threads on a cloud Linux box and is expecting ~300 tok/s soon. 

Further, he said that If you can get 7B model to run at nice and interactive rates then you can go from “scratch-trained micromodels” to “LoRA fine tuned 7B base model”, all within the code of the minimal llama2.c repo (both training and inference). Can reach more capability and with less training data.

Interestingly, the success of Karpathy’s approach lies in its ability to achieve highly interactive rates, even with reasonably sized models containing a few million parameters and trained on a 15 million parameter model of TinyStories dataset. 

Hopefully it will bring back the actual OpenAI which was started as an open source non-profit company where Karpathy was one of the initial founding members who played an active role in contributing to the open source community.