LLMs finally Bloom with Petals

This BitTorrent-style running of large language models (LLMs) allows many times faster inference when compared to offloading on single systems, closer to 1 second per token. Parallel inference can reach hundreds of tokens per second.

Mohit Pandey

Published 2023-02-01T05:30:00.000+05:30Updated 2023-04-01T05:30:00.000+05:30

Key takeaways

Petals enables users to collaboratively run large language models, overcoming memory constraints typically faced on individual systems.

This BitTorrent-style approach allows for significantly faster inference times compared to traditional offloading methods.

The project primarily supports CUDA-enabled PyTorch and is currently available only for Linux users.

Incentives like 'bloom points' encourage users to contribute GPU resources, addressing potential privacy concerns during collaboration.

While fine-tuning open-sourced LLMs is challenging, Petals aims to democratise access and foster innovation in the field.

Even when large language models like BLOOM, PaLM, or GPT get open-sourced, fine-tuning and inferencing them on your system is a memory-heavy task. This might hinder developers from running these models on their systems and thus slow down innovation, leaving it in the hands of only big players.

BigScience Workshop released Petals, which allows users to run language models with more than 100 billion parameters at home by loading a small part of the model on your machine, and then collaborating with other people to run other parts of inference and fine-tuning.

Click here to check out the repository on GitHub.

This BitTorrent-style running of large language models allows many times faster inference when compared to offloading on single systems, closer to 1 second per token. Parallel inference can reach hundreds of tokens per second.

The script is built for CUDA-enabled PyTorch and uses Anaconda to install and is only available for Linux users for now.

Mentioned in the GitHub page, “Petals” is a metaphor for a single person serving different parts of the model, and hosting together the entire language model - BLOOM, which has 176 billion parameters.

Since the collaboration might be slow in the beginning because of privacy or security issues, the team has decided to give “bloom points” as an incentive system for people who donate their GPU time for people to fine tune it.

Also read: ChatGPT and DALL-E on Discord

LLMs finally Bloom with Petals

Key takeaways

Stay ahead with AI insights.