Repolode
Submit
microsoft

microsoft/onnxruntime-genai

A C/C++ library with Python, C#, and Java bindings for running generative AI models on ONNX Runtime, handling the full LLM generation loop.

View on GitHub
Stars
1,153
Forks
383
Open issues
276
Last commit
4 hours ago

What is it?

ONNX Runtime GenAI is a library for running generative AI models with ONNX Runtime. It implements the generation loop for ONNX models, covering pre- and post-processing, inference, logits processing, search and sampling, KV cache management, and grammar specification for tool calling.

The project supports a broad set of model families, including Llama, Phi, Qwen, Gemma, Mistral, DeepSeek, and Whisper, and exposes APIs for Python, C#, and C/C++, with Java available when built from source. It runs on Linux, Windows, Mac, and Android, and can use CPU, CUDA, DirectML, OpenVINO, QNN, WebGPU, and other accelerators.

The library also provides multi-LoRA adapters, continuous decoding, and constrained decoding. The README notes that it powers Foundry Local, Windows ML, and the Visual Studio Code AI Toolkit.

Key features

  • 01Implements the full generation loop for ONNX LLMs, including sampling and KV cache management.
  • 02Offers Python, C#, and C/C++ APIs, with Java via source build.
  • 03Runs on Linux, Windows, Mac, and Android across x86, x64, and arm64.
  • 04Supports hardware acceleration through CUDA, DirectML, OpenVINO, QNN, WebGPU, and others.
  • 05Provides multi-LoRA, continuous decoding, and constrained decoding.

Use cases

  • Running small language models such as Phi or Llama locally on a laptop or device.
  • Embedding LLM inference into a C# or Windows application via ONNX Runtime.
  • Deploying quantized models on NPUs or GPUs from different vendors on one codebase.
  • Enforcing structured output or tool-call grammars during generation.

Quick start

  1. 1

    Download the Phi-3 model from Hugging Face

    huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* --local-dir .
  2. 2

    Install numpy

    pip install numpy
  3. 3

    Install the onnxruntime-genai Python package

    pip install --pre onnxruntime-genai
  4. 4

    Save the README's Python sample as a script and run it to generate text

Things to know

  • Java bindings require building from source.
  • Examples on the main branch may not match the latest stable release.

Topics

#onnx#llm#inference#on-device-ai#generative-ai#model-serving#quantization

FAQ

What is onnxruntime-genai?
A C/C++ library with Python, C#, and Java bindings for running generative AI models on ONNX Runtime, handling the full LLM generation loop. ONNX Runtime GenAI is a library for running generative AI models with ONNX Runtime. It implements the generation loop for ONNX models, covering pre- and post-processing, inference, logits processing, search and sampling, KV cache management, and grammar specification for tool calling.
Is onnxruntime-genai free to use? What license does it have?
onnxruntime-genai is open source under the MIT license. Check the license terms before using it in commercial products.
Is onnxruntime-genai still maintained?
Actively maintained. The last commit to onnxruntime-genai was on Nov 7, 2026.
What can I use onnxruntime-genai for?
Running small language models such as Phi or Llama locally on a laptop or device. Embedding LLM inference into a C# or Windows application via ONNX Runtime. Deploying quantized models on NPUs or GPUs from different vendors on one codebase. Enforcing structured output or tool-call grammars during generation.
How do I install onnxruntime-genai?
Download the Phi-3 model from Hugging Face: huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* --local-dir . Install numpy: pip install numpy Install the onnxruntime-genai Python package: pip install --pre onnxruntime-genai Save the README's Python sample as a script and run it to generate text
Where can I get onnxruntime-genai?
The source code and installation instructions are on GitHub: https://github.com/microsoft/onnxruntime-genai

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects