Repolode
Submit
huggingface

huggingface/optimum-quanto

A PyTorch quantization backend for Hugging Face Optimum that quantizes model weights and activations to int2 through float8.

View on GitHub
Stars
1,056
Forks
100
Open issues
5
Last commit
12 days ago

What is it?

Optimum Quanto is a PyTorch quantization backend that serves as part of the Hugging Face Optimum ecosystem. It converts float models into lower-bitwidth versions, supporting int2, int4, int8 and float8 weights as well as int8 and float8 activations, which reduces device memory roughly in proportion to the bit width.

The library works in eager mode, so it can handle models that are not traceable, and quantized models can run on CUDA, MPS and other devices. It offers helper classes for quantizing and saving Hugging Face LLMs and diffusers pipeline components, plus a low-level API for vanilla PyTorch models covering quantization, calibration, quantization-aware tuning, freezing and serialization via safetensors.

The project is in maintenance mode, so only minor fixes and documentation changes are being accepted. Its README points to bitsandbytes and torchAO for production-oriented or actively developed quantization work. Torch compiler (dynamo) compatibility is listed as not yet implemented.

Key features

  • 01Supports int2, int4, int8 and float8 weights, and int8 and float8 activations.
  • 02Runs in eager mode, so non-traceable models are supported.
  • 03Quantized models can be placed on CUDA, MPS and other devices.
  • 04Provides helper classes to quantize, save and reload Hugging Face LLMs and diffusers components.
  • 05Serializes to pytorch weight_only format and safetensors.

Use cases

  • Shrinking large Hugging Face LLMs to fit limited GPU memory.
  • Quantizing diffusers transformer components such as PixArt for lower-memory image generation.
  • Applying quantization-aware training to vanilla PyTorch models with the low-level API.

In collections

AI image generators & Stable Diffusion tools

Topics

#pytorch#quantization#llm#diffusion-models#model-compression#safetensors#hugging-face#optimum

FAQ

What is optimum-quanto?
A PyTorch quantization backend for Hugging Face Optimum that quantizes model weights and activations to int2 through float8. Optimum Quanto is a PyTorch quantization backend that serves as part of the Hugging Face Optimum ecosystem. It converts float models into lower-bitwidth versions, supporting int2, int4, int8 and float8 weights as well as int8 and float8 activations, which reduces device memory roughly in proportion to the bit width.
Is optimum-quanto free to use? What license does it have?
optimum-quanto is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is optimum-quanto still maintained?
Actively maintained. The last commit to optimum-quanto was on Oct 26, 2026.
What can I use optimum-quanto for?
Shrinking large Hugging Face LLMs to fit limited GPU memory. Quantizing diffusers transformer components such as PixArt for lower-memory image generation. Applying quantization-aware training to vanilla PyTorch models with the low-level API.
Where can I get optimum-quanto?
The source code and installation instructions are on GitHub: https://github.com/huggingface/optimum-quanto

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects