pytorch/ao

What is it?

Key features

• 01Int4, int8 and float8 weight and activation quantization for inference.
• 02Quantization-aware training (QAT) recipes that recover accuracy lost after quantization.
• 03Float8 and MXFP8 training support that works with torch.compile and FSDP2.
• 04Quantized optimizers (8-bit, 4-bit, fp8 AdamW) and single-GPU CPU offloading to cut memory use.
• 05Integrations with Hugging Face transformers, diffusers, vLLM, SGLang and ExecuTorch.

Use cases

• Serving a large language model with int4 or float8 weights to reduce GPU memory and latency.
• Fine-tuning a quantized model with QAT to limit accuracy loss from low-bit inference.
• Running float8 pre-training of large transformer models on multi-GPU clusters.
• Deploying a quantized model to a mobile or edge device through ExecuTorch.

Quick start

1. 1

   Install the latest stable TorchAO from PyPI

   pip install torchao
2. 2

   Quantize model weights to int4 using quantize_ (replace model with your PyTorch model)

   quantize_(model, Int4WeightOnlyConfig(group_size=32, int4_packing_format="tile_packed_to_4d", int4_choose_qparams_algorithm="hqq"))

Things to know

• !Some optional kernels and training features need extra packages and specific CUDA builds.
• !Float8 and MoE training features depend on particular GPU hardware.

Topics

FAQ

What is ao?

PyTorch-native library for quantization, sparsity and float8 training, covering both model training and inference serving. TorchAO is a quantization and model optimization library built directly on PyTorch. It provides weight-only and dynamic quantization (including int4, int8 and float8), quantization-aware training (QAT), float8 and MXFP8 training recipes, sparsity, and quantized optimizers that reduce optimizer memory use.

Is ao free to use? What license does it have?

ao is open source under the Other license. Check the license terms before using it in commercial products.

Is ao still maintained?

Actively maintained. The last commit to ao was on Oct 19, 2026.

What can I use ao for?

Serving a large language model with int4 or float8 weights to reduce GPU memory and latency. Fine-tuning a quantized model with QAT to limit accuracy loss from low-bit inference. Running float8 pre-training of large transformer models on multi-GPU clusters. Deploying a quantized model to a mobile or edge device through ExecuTorch.

How do I install ao?

Install the latest stable TorchAO from PyPI: pip install torchao Quantize model weights to int4 using quantize_ (replace model with your PyTorch model): quantize_(model, Int4WeightOnlyConfig(group_size=32, int4_packing_format="tile_packed_to_4d", int4_choose_qparams_algorithm="hqq"))

Where can I get ao?

The source code and installation instructions are on GitHub: https://github.com/pytorch/ao