Repolode
Submit
NVIDIA

NVIDIA/Model-Optimizer

NVIDIA's open-source library for compressing deep learning models via quantization, pruning, distillation, sparsity and speculative decoding to speed up inference.

View on GitHubWebsite
Stars
5,553
Forks
803
Open issues
524
Last commit
3 hours ago

What is it?

NVIDIA Model Optimizer (ModelOpt) is a Python library that applies model optimization techniques such as post-training quantization, quantization-aware training, pruning, knowledge distillation, Neural Architecture Search, sparsity and speculative decoding. It takes Hugging Face, PyTorch or ONNX models as input and provides Python APIs to compose these techniques and export an optimized quantized checkpoint.

The exported checkpoints are intended for downstream inference frameworks including TensorRT-LLM, TensorRT, vLLM and SGLang. The project also integrates with Megatron-LM, Megatron-Bridge and Hugging Face Accelerate for training-time optimization, and its Hugging Face export API covers both transformers and diffusers models.

The repository includes examples for LLMs, VLMs, diffusion models and ONNX, along with a support matrix per model type and a set of pre-quantized checkpoints published on Hugging Face.

Key features

  • 01Post-training quantization and quantization-aware training, including NVFP4 and FP8 formats
  • 02Structured pruning and knowledge distillation for shrinking LLMs
  • 03Speculative decoding and sparsity techniques
  • 04Exports quantized checkpoints for TensorRT-LLM, vLLM and SGLang
  • 05Supports Hugging Face, PyTorch and ONNX model inputs

Use cases

  • Quantizing a large Hugging Face LLM to NVFP4 or FP8 before serving it with vLLM or TensorRT-LLM
  • Pruning and distilling a larger model into a smaller one that retains most of its quality
  • Adding a draft module to speed up token generation with speculative decoding
  • Reducing diffusion model latency for image or video generation pipelines

Quick start

  1. 1

    Install the stable release from PyPI

    pip install -U nvidia-modelopt[all]

Things to know

  • Installing pulls in additional third-party open source projects whose license terms you should review.
  • The NVIDIA container images it can be used with have their own license terms that you must review before use.

Topics

#model-compression#quantization#pruning#knowledge-distillation#inference-optimization#speculative-decoding#tensorrt#llm

FAQ

What is Model-Optimizer?
NVIDIA's open-source library for compressing deep learning models via quantization, pruning, distillation, sparsity and speculative decoding to speed up inference. NVIDIA Model Optimizer (ModelOpt) is a Python library that applies model optimization techniques such as post-training quantization, quantization-aware training, pruning, knowledge distillation, Neural Architecture Search, sparsity and speculative decoding. It takes Hugging Face, PyTorch or ONNX models as input and provides Python APIs to compose these techniques and export an optimized quantized checkpoint.
Is Model-Optimizer free to use? What license does it have?
Model-Optimizer is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is Model-Optimizer still maintained?
Actively maintained. The last commit to Model-Optimizer was on Nov 7, 2026.
What can I use Model-Optimizer for?
Quantizing a large Hugging Face LLM to NVFP4 or FP8 before serving it with vLLM or TensorRT-LLM Pruning and distilling a larger model into a smaller one that retains most of its quality Adding a draft module to speed up token generation with speculative decoding Reducing diffusion model latency for image or video generation pipelines
How do I install Model-Optimizer?
Install the stable release from PyPI: pip install -U nvidia-modelopt[all]
Where can I get Model-Optimizer?
The source code and installation instructions are on GitHub: https://github.com/NVIDIA/Model-Optimizer

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects