Even when large language models like BLOOM, PaLM, or GPT get open-sourced, fine-tuning and inferencing them on your system is a memory-heavy task. This might hinder developers from running these models on their systems and thus slow down innovation, leaving it in the hands of only big players.

BigScience Workshop released Petals, which allows users to run language models with more than 100 billion parameters at home by loading a small part of the model on your machine, and then collaborating with other people to run other parts of inference and fine-tuning. 

Click here to check out the repository on GitHub.

This BitTorrent-style running of large language models allows many times faster inference when compared to offloading on single systems, closer to 1 second per token. Parallel inference can reach hundreds of tokens per second.

The script is built for CUDA-enabled PyTorch and uses Anaconda to install and is only available for Linux users for now.

Mentioned in the GitHub page, “Petals” is a metaphor for a single person serving different parts of the model, and hosting together the entire language model – BLOOM, which has 176 billion parameters.

Since the collaboration might be slow in the beginning because of privacy or security issues, the team has decided to give “bloom points” as an incentive system for people who donate their GPU time for people to fine tune it. 

Also read: ChatGPT and DALL-E on Discord