Repolode
Submit
vllm-project

vllm-project/llm-compressor

A Python library for compressing LLMs through quantization and pruning, producing checkpoints that load directly into vLLM for inference.

View on GitHubWebsite
Stars
3,901
Forks
707
Open issues
124
Last commit
22 minutes ago

What is it?

LLM Compressor is a library for optimizing large language models before deployment with vLLM. It applies quantization to weights, activations, KV cache and attention, and can also apply REAP expert pruning to mixture-of-experts models. Models are saved in the compressed-tensors format, which vLLM can load without extra conversion.

The library works with Hugging Face models and repositories. It offers several algorithms, including simple round-to-nearest PTQ, GPTQ, AWQ, SmoothQuant, AutoRound and rotation-based methods such as SpinQuant and QuIP. Supported precisions range from FP8, INT8 and NVFP4/MXFP4 microscale formats to 4-bit and 8-bit weight-only schemes.

For very large models, it supports DDP quantization, sequential onloading and disk offloading, so compression can run on limited hardware. Its API is a single `oneshot` call that takes a recipe, which keeps the workflow short for common cases.

Key features

  • 01Weight, activation, KV cache and attention quantization with formats such as FP8, INT8, NVFP4 and MXFP4.
  • 02Algorithms including GPTQ, AWQ, SmoothQuant, AutoRound and round-to-nearest PTQ.
  • 03Outputs compressed-tensors checkpoints that vLLM loads directly.
  • 04Works with Hugging Face models and repositories.
  • 05DDP, sequential onloading and disk offloading for compressing very large models.
  • 06REAP expert pruning for mixture-of-experts models.

Use cases

  • Shrinking a model's VRAM footprint so it fits on fewer GPUs for vLLM serving.
  • Producing FP4 or FP8 checkpoints of new open-weight models for production inference.
  • Compressing very large models on limited hardware using disk offloading or DDP.
  • Experimenting with different quantization algorithms and schemes via a recipe.

Topics

#llm#quantization#model-compression#vllm#inference#gptq#awq#fp8#compression

FAQ

What is llm-compressor?
A Python library for compressing LLMs through quantization and pruning, producing checkpoints that load directly into vLLM for inference. LLM Compressor is a library for optimizing large language models before deployment with vLLM. It applies quantization to weights, activations, KV cache and attention, and can also apply REAP expert pruning to mixture-of-experts models. Models are saved in the compressed-tensors format, which vLLM can load without extra conversion.
Is llm-compressor free to use? What license does it have?
llm-compressor is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is llm-compressor still maintained?
Actively maintained. The last commit to llm-compressor was on Oct 22, 2026.
What can I use llm-compressor for?
Shrinking a model's VRAM footprint so it fits on fewer GPUs for vLLM serving. Producing FP4 or FP8 checkpoints of new open-weight models for production inference. Compressing very large models on limited hardware using disk offloading or DDP. Experimenting with different quantization algorithms and schemes via a recipe.
Where can I get llm-compressor?
The source code and installation instructions are on GitHub: https://github.com/vllm-project/llm-compressor

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects