Microsoft, ETH Zurich Introduce SliceGPT For Compressing LLMs

Researchers from Microsoft and ETH Zurich have unveiled SliceGPT, a post-training sparsification scheme aimed at addressing the substantial computational and memory costs associated with large language models. 

Click here to read the research paper. 

The research demonstrates how SliceGPT efficiently reduces the size of language models by eliminating up to 25% of model parameters, including embeddings, while maintaining impressive zero-shot task performance.

The study focuses on three prominent models: LLAMA2-70B, OPT 66B, and Phi-2. For LLAMA2-70B, SliceGPT achieves a remarkable reduction to 64% of the dense model's total compute for inference on 24GB consumer GPUs. 

Similarly, on 40GB A100 GPUs, SliceGPT decreases total compute to 66% for LLAMA2-70B. The research also showcases that SliceGPT requires fewer GPUs to operate, resulting in enhanced speed and efficiency without the need for additional code optimisation.

The key innovation behind SliceGPT lies in its ability to replace each weight matrix with a smaller, dense matrix, effectively reducing the embedding dimension of the network. 

The sparsification technique offers a compelling solution to resource constraints associated with large language models, showcasing its potential to pave the way for future advancements in reducing memory and computation demands for pre-trained models.

Microsoft's SliceGPT not only proves its efficiency in model size reduction but also introduces a new perspective—computational invariance in transformer networks. This insight is expected to inspire and unlock new avenues for further developments in the realm of pre-trained models.