Is EleutherAI Closely Following OpenAI’s Route?

EleutherAI, which began as an open-source AI research group on a Discord server in July 2020 by a group of hackers—namely, Connor Leahy, Sid Black, and Leo Gao—has announced that it is becoming a non-profit research institute. This step is seen as an encouraging move towards promoting open-source AI platforms and EleutherAI’s Discord server will continue to serve as an unrestricted platform for users. Stella Rose Biderman, an AI researcher at EleutherAI, has referred to it as a “research institute with open doors”. In 2023, the institute plans to focus more on alignment and interpretability projects.

Connor Leahy, the founder of EleutherAI, has announced that he and, his friend and co-founder, Sid Black, will step down as organisers and leaders of the group to focus on his AI alignment research startup, ‘Conjecture‘. Two other researchers with the institute, Louis Castricato and Tanishq Abraham, have also left to pursue their individual projects. Castricato has started ‘CarperAI’, which focuses on preference learning and RLHF, and Abraham has launched ‘MedARC’, which focuses on biomedical applications of AI technologies.

EleutherAI said that until now contributors were mostly trying to balance their regular forty-hour workweek and work on AI technology on the side. With the move of becoming a non-profit research institute, over twenty of their regular contributors will now be working full-time on their research activities.  

EleutherAI has OpenAI connection

In June 2020, OpenAI launched GPT-3 which got the ML community buzzing. During that time, on Shawn Presser’s discord server, Connor Leahy took up a paper to challenge the scaling of neural networks and thus led to an active community that focused on promoting open source AI. 

Connor used his access to Google’s TRC TPU Research Cloud to work with fellow hackers to see how far they could go with the research. TRC enables researchers to apply for access to clusters of more than 1000 Cloud TPU devices. Once accepted, researchers can access Cloud TPUs at no extra charge.

EleutherAI has built 825 GB of language modelling dataset called ‘The Pile’, which is curated from datasets including arXiv, GitHub, Wikipedia, StackExchange and HackerNews. They then built GPT-J which is a 6-billion-parameter model trained on The Pile. 

In the last one-and-a-half years, EleutherAI members have authored 28 papers, trained dozens of models, and released 10 codebases. 

Backup Partners

By transforming into a not-for-profit research institute, EleutherAI will continue to thrive on the vast domain expertise in the platform. The company has support from some of the major players in the AI market. Stability AI, Hugging Face, CoreWeave, Canva, Google TRC, Nat Friedman (former CEO of GitHub) and Lambda Labs are among the prominent collaborators for the platform. 

‘CoreWeave’, a specialised cloud service provider and part of NVIDIA Preferred Cloud Services Provider network, are the GPU providers for EleutherAI to train GPT-3 language models. CoreWeave was not only keen about open source code but also about “breaking Microsoft’s monopoly”. Built on CoreWeave GPU, EleutherAI launched its largest LLM GPT-NeoX-20B. 

EleutherAI, along with CompVis LMU, Runway and LAION, helped StabilityAI create their text-to-image AI system, ‘Stable Diffusion’. Since then, StabilityAI has given a portion of compute from its AWS cluster for EleutherAI’s research. 

Gary Marcus has always been a vocal supporter of EleutherAI and their vision to provide open source software. 

“Non-profit” AI future

The company insists that EleutherAI will continue to work the way it did by focusing on fostering an open source AI community. However, going by the recent example of OpenAI’s functioning, a once non-profit organisation that was quick to go down the route of profitability by shedding the tag and becoming a “capped profit” structure, where investors can get profits which are capped at 100 times their investment value. 

Such a transition from a non-profit tag is likely to happen when a company’s funds dry up and that is something EleutherAI will need to be careful about. Considering the kind of companies backing the research institute, the problem may not arise. However, those same companies are commercially motivated ventures and whether they will ultimately affect the business goals of EleutherAI is something that only time will tell.