Intel's open-source Python library for compressing deep learning models via quantization, sparsity and distillation across PyTorch, JAX and ONNX.
2.7K331PythonApache-2.0Updated 23 hours ago
openvinotoolkit/nncf
A Python framework for compressing neural networks via quantization, pruning and sparsity to speed up inference in OpenVINO, PyTorch and ONNX.
- Stars
- 1,211
- Forks
- 309
- Open issues
- 50
- Last commit
- 2 days ago
What is it?
NNCF (Neural Network Compression Framework) is a Python package from the OpenVINO toolkit that reduces the size and cost of neural network inference while aiming to keep accuracy loss small. It offers post-training algorithms such as 8-bit Post-Training Quantization and LLM weight compression, which need only a model and a small calibration dataset, as well as training-time methods including Quantization Aware Training, LoRA-based weight-only quantization and pruning.
NNCF accepts models from PyTorch, TorchFX, ONNX and OpenVINO, with OpenVINO as the preferred backend for post-training quantization. Its API is shared across algorithms, and it can export compressed PyTorch models to ONNX or TensorFlow formats for use with OpenVINO.
The project is used as a compression backend by HuggingFace Optimum Intel, Ultralytics, ExecuTorch and torch.compile's OpenVINO integration, and ships with notebooks and example scripts covering image classification, object detection, segmentation and LLMs.
Key features
- 01Post-training 8-bit quantization that needs only a small calibration dataset.
- 02Weight compression for LLMs, including hyperparameter search and scale estimation.
- 03Quantization-aware training and pruning for PyTorch models.
- 04Common API across OpenVINO, PyTorch, TorchFX and ONNX backends.
- 05Export of compressed PyTorch models to ONNX or TensorFlow formats.
Use cases
- Shrinking a vision or NLP model before deploying it with OpenVINO on Intel hardware.
- Compressing LLM weights to INT4 for lighter local inference.
- Recovering accuracy after quantization with quantization-aware fine-tuning in PyTorch.
- Adding quantization to an existing ONNX or PyTorch export pipeline.
Quick start
- 1
Install NNCF from PyPI, following the Installation guide in the README
- 2
Prepare your model and a small calibration dataset (about 300 samples)
- 3
Run nncf.quantize on your model with the calibration dataset, as shown in the Usage section
Things to know
- The README badges list Python 3.11+ as the required version.
- TorchFX support for post-training compression and activation sparsity in PyTorch are marked experimental.
Topics
FAQ
- What is nncf?
- A Python framework for compressing neural networks via quantization, pruning and sparsity to speed up inference in OpenVINO, PyTorch and ONNX. NNCF (Neural Network Compression Framework) is a Python package from the OpenVINO toolkit that reduces the size and cost of neural network inference while aiming to keep accuracy loss small. It offers post-training algorithms such as 8-bit Post-Training Quantization and LLM weight compression, which need only a model and a small calibration dataset, as well as training-time methods including Quantization Aware Training, LoRA-based weight-only quantization and pruning.
- Is nncf free to use? What license does it have?
- nncf is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
- Is nncf still maintained?
- Actively maintained. The last commit to nncf was on Oct 19, 2026.
- What can I use nncf for?
- Shrinking a vision or NLP model before deploying it with OpenVINO on Intel hardware. Compressing LLM weights to INT4 for lighter local inference. Recovering accuracy after quantization with quantization-aware fine-tuning in PyTorch. Adding quantization to an existing ONNX or PyTorch export pipeline.
- How do I install nncf?
- Install NNCF from PyPI, following the Installation guide in the README Prepare your model and a small calibration dataset (about 300 samples) Run nncf.quantize on your model with the calibration dataset, as shown in the Usage section
- Where can I get nncf?
- The source code and installation instructions are on GitHub: https://github.com/openvinotoolkit/nncf
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
Ultralytics YOLOv5, a PyTorch-based computer vision toolkit for object detection, instance segmentation and image classification with training and export.
58.2K17.5KPythonAGPL-3.0Updated 3 days ago
A deep learning optimization library for making distributed training and inference of large models easier and more efficient.
43.2K5KPythonApache-2.0Updated 2 days ago
Python library that runs large language models layer by layer so 70B-class models can infer on a single 4GB GPU, without quantization.
39.5K4KJupyter NotebookApache-2.0Updated 2 days ago
A high-performance neural network inference framework in C++ for running deep learning models on mobile, embedded, and desktop devices.
24K4.5KC++OtherUpdated yesterday
PyTorch-native library for quantization, sparsity and float8 training, covering both model training and inference serving.
3K655PythonOtherUpdated 2 days ago