Repolode
Submit
intel

intel/neural-compressor

Intel's open-source Python library for compressing deep learning models via quantization, sparsity and distillation across PyTorch, JAX and ONNX.

View on GitHubWebsite
Stars
2,722
Forks
331
Open issues
30
Last commit
22 hours ago

What is it?

Intel Neural Compressor is a Python library for shrinking and speeding up deep learning models. It provides post-training and quantization-aware methods such as static and dynamic quantization, SmoothQuant, weight-only quantization, mixed precision and FP8, along with low-bit formats like INT4, MXFP4 and NVFP4 aimed at large language models and vision-language models.

The library ships framework extensions for PyTorch and JAX, with a transformers-style API for loading and quantizing models. Quantization of LLMs and VLMs is integrated with AutoRound. Documented hardware targets include Intel Gaudi AI accelerators, Intel Core Ultra, Xeon Scalable processors and Intel Data Center GPUs, with limited testing on AMD CPUs, ARM CPUs and NVIDIA GPUs.

The project also includes an auto-tuning module that searches for quantization configurations meeting an accuracy target, and documentation covers GPTQ and AWQ style weight-only workflows.

Key features

  • 01Static, dynamic and weight-only quantization for PyTorch and JAX models
  • 02SmoothQuant and quantization-aware training support
  • 03FP8, MXFP8/MXFP4 and NVFP4 low-precision formats (some experimental)
  • 04Transformers-like API for quantizing LLMs and VLMs via AutoRound
  • 05Auto-tuning to find quantization settings that meet an accuracy target

Use cases

  • Reducing memory and latency of large language models for inference on Intel hardware
  • Quantizing PyTorch or JAX vision models before deployment
  • Applying FP8 quantization on Intel Gaudi accelerators

Quick start

  1. 1

    Install the PyTorch backend package from PyPI

    pip install neural-compressor-pt
  2. 2

    Install the JAX backend package from PyPI (optional)

    pip install neural-compressor-jax
  3. 3

    Run the FP8 quantization example code (requires Intel Gaudi2 hardware or the Gaudi Docker image)

Things to know

  • The main getting-started example targets Intel Gaudi2 hardware, which most developers will not have.
  • Some features are only experimental, such as NVFP4, MXFP8/MXFP4 and FP8 for Keras/JAX.
  • Support for AMD CPU, ARM CPU and NVIDIA GPU is limited testing only.
  • Hardware-specific setup, such as matching Gaudi software versions, adds installation effort.

Topics

#quantization#model-compression#llm#pytorch#int4#fp8#pruning#deep-learning#auto-tuning#awq

FAQ

What is neural-compressor?
Intel's open-source Python library for compressing deep learning models via quantization, sparsity and distillation across PyTorch, JAX and ONNX. Intel Neural Compressor is a Python library for shrinking and speeding up deep learning models. It provides post-training and quantization-aware methods such as static and dynamic quantization, SmoothQuant, weight-only quantization, mixed precision and FP8, along with low-bit formats like INT4, MXFP4 and NVFP4 aimed at large language models and vision-language models.
Is neural-compressor free to use? What license does it have?
neural-compressor is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is neural-compressor still maintained?
Actively maintained. The last commit to neural-compressor was on Oct 20, 2026.
What can I use neural-compressor for?
Reducing memory and latency of large language models for inference on Intel hardware Quantizing PyTorch or JAX vision models before deployment Applying FP8 quantization on Intel Gaudi accelerators
How do I install neural-compressor?
Install the PyTorch backend package from PyPI: pip install neural-compressor-pt Install the JAX backend package from PyPI (optional): pip install neural-compressor-jax Run the FP8 quantization example code (requires Intel Gaudi2 hardware or the Gaudi Docker image)
Where can I get neural-compressor?
The source code and installation instructions are on GitHub: https://github.com/intel/neural-compressor

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects