Repolode
Submit
intel

intel/auto-round

An LLM and VLM post-training quantization toolkit from Intel that compresses models to low-bit weights for Transformers, vLLM and SGLang.

View on GitHub
Stars
1,684
Forks
193
Open issues
109
Last commit
3 hours ago

What is it?

AutoRound is a post-training quantization toolkit for large language models and vision-language models. It uses signed-gradient optimization of weight rounding (the SignRound approach) to keep accuracy at low bit widths, and also offers an adaptive mixed-bit scheme generator called AutoScheme.

It supports weight-only schemes such as W4A16, W2A16 and W8A16, along with MXFP4, NVFP4 and FP8 data types, and can export to its own auto_round format as well as AutoAWQ, AutoGPTQ, llm-compressor and GGUF. Quantized models can be loaded in Transformers, vLLM and SGLang, and the project targets Intel CPUs and XPUs, Gaudi accelerators and NVIDIA GPUs.

Users pick a recipe from fast RTN-style modes to a higher-accuracy setting, and can run it from a CLI or the Python API with configurable calibration datasets, iterations and memory options.

Key features

  • 01Weight-only quantization at W2 to W8 bit widths, plus MXFP4, NVFP4 and FP8 types.
  • 02Exports to auto_round, AutoAWQ, AutoGPTQ, llm-compressor and GGUF formats.
  • 03AutoScheme generates mixed-bit quantization recipes under a target average bit-width.
  • 04Supports quantizing the text module of vision-language models.
  • 05Multi-GPU quantization and selectable calibration datasets.

Use cases

  • Shrinking a 7B-70B LLM to 4-bit weights for cheaper inference on a single GPU.
  • Preparing quantized checkpoints to serve through vLLM or SGLang.
  • Producing GGUF files for llama.cpp-compatible runtimes.
  • Experimenting with mixed-precision recipes to hit a memory budget.

Quick start

  1. 1

    Install auto-round from PyPI (CPU/CUDA)

    pip install auto-round
  2. 2

    Quantize a model with the CLI

    auto-round --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round" --output_dir ./tmp_autoround

Things to know

  • Some features are experimental, such as algorithm composition and the MXFP4 scheme, which has no real kernels.
  • Recent releases note that some optimizations may introduce regressions or issues.
  • Best-accuracy recipes are several times slower than the default.

Topics

#quantization#llm#vlm#int4#post-training-quantization#transformers#vllm#deep-learning#diffusers#gguf

FAQ

What is auto-round?
An LLM and VLM post-training quantization toolkit from Intel that compresses models to low-bit weights for Transformers, vLLM and SGLang. AutoRound is a post-training quantization toolkit for large language models and vision-language models. It uses signed-gradient optimization of weight rounding (the SignRound approach) to keep accuracy at low bit widths, and also offers an adaptive mixed-bit scheme generator called AutoScheme.
Is auto-round free to use? What license does it have?
auto-round is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is auto-round still maintained?
Actively maintained. The last commit to auto-round was on Oct 21, 2026.
What can I use auto-round for?
Shrinking a 7B-70B LLM to 4-bit weights for cheaper inference on a single GPU. Preparing quantized checkpoints to serve through vLLM or SGLang. Producing GGUF files for llama.cpp-compatible runtimes. Experimenting with mixed-precision recipes to hit a memory budget.
How do I install auto-round?
Install auto-round from PyPI (CPU/CUDA): pip install auto-round Quantize a model with the CLI: auto-round --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round" --output_dir ./tmp_autoround
Where can I get auto-round?
The source code and installation instructions are on GitHub: https://github.com/intel/auto-round

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects