Repolode
Submit
microsoft

microsoft/Olive

Python toolkit that composes model finetuning, conversion, and quantization steps to produce optimized ONNX models for CPUs, GPUs, and NPUs.

View on GitHubWebsite
Stars
2,409
Forks
326
Open issues
87
Last commit
23 hours ago

What is it?

Olive is an AI model optimization toolkit built around the ONNX Runtime. Given a model and a target hardware platform, it chains together optimization techniques such as quantization and graph optimization, and aims to output efficient ONNX models for inference on cloud or edge devices while respecting constraints like accuracy and latency.

The project provides an `olive optimize` command-line entry point that takes a Hugging Face model name and a target precision, for example int4, and runs the full pipeline: downloading the model, quantizing it with GPTQ, exporting the ONNX graph with weights, and optimizing that graph. Popular architectures such as Llama, Phi, Qwen, and Gemma are supported out of the box, and other architectures can be handled by describing their inputs and outputs.

Models produced by Olive are intended to run on ONNX Runtime, which has bindings for Python, C/C++, C#, Java, and JavaScript. Recent releases mention NPU support and additional quantization algorithms.

Key features

  • 01Automatic optimization pipeline driven by a single olive optimize command.
  • 02Quantization to low precisions such as int4 using GPTQ.
  • 03Exports Hugging Face models to ONNX graphs with separate weight files.
  • 04Targets CPUs, GPUs, and NPUs through ONNX Runtime.
  • 05Supports finetuning and conversion steps alongside optimization.

Use cases

  • Converting a small language model to int4 ONNX format for on-device inference.
  • Preparing Hugging Face LLMs for deployment with ONNX Runtime on Windows or edge hardware.
  • Experimenting with quantization settings to balance model size, accuracy, and latency.

Quick start

  1. 1

    Install Olive with pip

    pip install olive-ai
  2. 2

    Install transformers and onnxruntime-genai

    pip install transformers onnxruntime-genai
  3. 3

    Run automatic optimization of Qwen2.5-0.5B-Instruct to int4

    olive optimize --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct --precision int4 --output_path models/qwen

Things to know

  • Requires a Python environment and downloads of models from Hugging Face.
  • Some optimization features need optional extras and additional dependencies.

Topics

#onnx#model-optimization#quantization#onnx-runtime#llm#inference#fine-tuning

FAQ

What is Olive?
Python toolkit that composes model finetuning, conversion, and quantization steps to produce optimized ONNX models for CPUs, GPUs, and NPUs. Olive is an AI model optimization toolkit built around the ONNX Runtime. Given a model and a target hardware platform, it chains together optimization techniques such as quantization and graph optimization, and aims to output efficient ONNX models for inference on cloud or edge devices while respecting constraints like accuracy and latency.
Is Olive free to use? What license does it have?
Olive is open source under the MIT license. Check the license terms before using it in commercial products.
Is Olive still maintained?
Actively maintained. The last commit to Olive was on Nov 7, 2026.
What can I use Olive for?
Converting a small language model to int4 ONNX format for on-device inference. Preparing Hugging Face LLMs for deployment with ONNX Runtime on Windows or edge hardware. Experimenting with quantization settings to balance model size, accuracy, and latency.
How do I install Olive?
Install Olive with pip: pip install olive-ai Install transformers and onnxruntime-genai: pip install transformers onnxruntime-genai Run automatic optimization of Qwen2.5-0.5B-Instruct to int4: olive optimize --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct --precision int4 --output_path models/qwen
Where can I get Olive?
The source code and installation instructions are on GitHub: https://github.com/microsoft/Olive

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects