Intel's open-source Python library for compressing deep learning models via quantization, sparsity and distillation across PyTorch, JAX and ONNX.
2.7K332PythonApache-2.0Updated yesterday
microsoft/TransformerCompression
SliceGPT is a post-training compression method that shrinks transformer models like LLMs by slicing rows and columns from weight matrices.
- Stars
- 465
- Forks
- 59
- Open issues
- 27
- Last commit
- 2 months ago
What is it?
SliceGPT makes transformer networks, including large language models, smaller after training. It first applies orthogonal transformations to each layer, which leave the model's outputs unchanged, and then removes the least-significant rows and columns of the weight matrices, chosen by eigenvalue decay. The model's structure stays the same, but the embedding dimension is reduced, which yields speedups without additional code optimization and a smaller memory footprint.
The repository contains the `slicegpt` Python package and scripts for reproducing the experiments from the ICLR'24 paper. Workflows include slicing a model, running recovery fine-tuning (LoRA-based), and evaluating with the LM Eval Harness. Supported models include Phi-2, Phi-3-mini, several Llama-2 and Llama-3 variants, and OPT models from 125M to 66B parameters.
New model families can be added by implementing a model adapter and a layer adapter, then updating the Hugging Face loading utility.
Key features
- 01Post-training compression that reduces the embedding dimension by slicing weight matrices.
- 02Orthogonal transformations per layer leave the model's function unchanged before slicing.
- 03Produces dense smaller weight matrices, so speedups need no custom kernels.
- 04Optional recovery fine-tuning with LoRA and Bayesian hyperparameter search.
- 05Evaluation integration with the LM Eval Harness.
- 06Adapter interface for adding new Hugging Face transformer architectures.
Use cases
- Reducing memory footprint of Phi-2 or Llama models for deployment on limited GPUs.
- Researching post-training compression of LLMs and reproducing the SliceGPT paper results.
- Adding SliceGPT support to a new transformer architecture via adapters.
Quick start
- 1
Install the slicegpt package with experiment extras
pip install -e .[experiment] - 2
From the experiments folder, compress microsoft/phi-2 with SliceGPT
python run_slicegpt.py --model microsoft/phi-2 --save-dir dir/to/save/sliced_model/in --sparsity 0.25 --device cuda:0 --eval-baseline --no-wandb
Things to know
- Only the Hugging Face models listed in the README are supported; other models need a new adapter.
- Some models require Hugging Face authentication via --hf-token or HF_TOKEN.
Topics
#llm#model-compression#pruning#transformer#pytorch#hugging-face#quantization-alternative#fine-tuning
FAQ
- What is TransformerCompression?
- SliceGPT is a post-training compression method that shrinks transformer models like LLMs by slicing rows and columns from weight matrices. SliceGPT makes transformer networks, including large language models, smaller after training. It first applies orthogonal transformations to each layer, which leave the model's outputs unchanged, and then removes the least-significant rows and columns of the weight matrices, chosen by eigenvalue decay. The model's structure stays the same, but the embedding dimension is reduced, which yields speedups without additional code optimization and a smaller memory footprint.
- Is TransformerCompression free to use? What license does it have?
- TransformerCompression is open source under the MIT license. Check the license terms before using it in commercial products.
- Is TransformerCompression still maintained?
- Updated within 6 months. The last commit to TransformerCompression was on Sep 10, 2026.
- What can I use TransformerCompression for?
- Reducing memory footprint of Phi-2 or Llama models for deployment on limited GPUs. Researching post-training compression of LLMs and reproducing the SliceGPT paper results. Adding SliceGPT support to a new transformer architecture via adapters.
- How do I install TransformerCompression?
- Install the slicegpt package with experiment extras: pip install -e .[experiment] From the experiments folder, compress microsoft/phi-2 with SliceGPT: python run_slicegpt.py --model microsoft/phi-2 --save-dir dir/to/save/sliced_model/in --sparsity 0.25 --device cuda:0 --eval-baseline --no-wandb
- Where can I get TransformerCompression?
- The source code and installation instructions are on GitHub: https://github.com/microsoft/TransformerCompression
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
A PyTorch quantization backend for Hugging Face Optimum that quantizes model weights and activations to int2 through float8.
1.1K99PythonApache-2.0Updated 11 days ago
A toolkit for merging pre-trained large language models by combining their weights with many merge algorithms, run on CPU or GPU.
7.4K+15 this week802PythonLGPL-3.0Updated 6 days ago
PyTorch DTensor-native training library for fine-tuning and pretraining LLMs, VLMs and diffusion models directly from Hugging Face checkpoints.
1.1K368PythonApache-2.0Updated 23 hours ago
NVIDIA's open-source library for compressing deep learning models via quantization, pruning, distillation, sparsity and speculative decoding to speed up inference.
AI & Machine Learning#model-compression#quantization#pruning#knowledge-distillation#inference-optimization
5.5K+70 this week800PythonApache-2.0Updated 22 hours ago
A Chinese-language open tutorial that teaches LLM principles by building a small LLaMA2-style model from scratch, covering Transformer, pretraining, fine-tuning, RAG and agents.
34.8K+135 this week3.3KJupyter NotebookOtherUpdated 3 months ago