Repolode
Submit
sonos

sonos/tract

A self-contained Rust inference engine that loads ONNX and NNEF models and runs them on CPU, Apple Metal, NVIDIA CUDA, and WebAssembly.

View on GitHub
Stars
3,111
Forks
298
Open issues
136
Last commit
20 hours ago

What is it?

tract is a neural-network inference engine written in Rust, developed at Sonos. It loads ONNX and NNEF models, optimises them, and runs them on targets ranging from embedded ARM CPUs to NVIDIA and Apple GPUs, in the browser via WebAssembly, or on Linux, macOS and Windows machines. A Python package built on the same core is also published on PyPI.

A central idea is translating a model once into NNEF-based intermediate format, tract-OPL, and then shipping only a small runtime with the application. The project also supports pulsified inference, which converts a sequence model so it processes fixed-size chunks along its streaming axis, serving both batch and low-latency real-time use such as wake-word detection and streaming speech recognition. Examples in the repository cover image classification, speech recognition, text generation with transformers, text-to-image generation, and face detection.

Key features

  • 01Loads ONNX, NNEF, TensorFlow Lite and legacy TensorFlow 1 frozen graph models.
  • 02Runs on CPU with hand-written SIMD kernels, plus Apple Metal and NVIDIA CUDA backends.
  • 03Compiles to WebAssembly for browser or WASI deployment.
  • 04Converts models to a compact NNEF-based format so runtime dependencies stay small.
  • 05Supports pulsified inference for streaming models such as wake-word and speech recognition.
  • 06Provides a Rust API and a Python package on PyPI.

Use cases

  • Running wake-word or streaming speech recognition on embedded ARM devices.
  • Shipping a small inference runtime without protobuf or training-framework loaders.
  • Running ONNX or NNEF models in a Rust application on CPU or GPU.
  • Executing image, speech or language models in the browser via WebAssembly.

Quick start

  1. 1

    Install the Python package from PyPI

    pip install tract
  2. 2

    Load an ONNX model and run it with the Rust API, following the README quick start example

Things to know

  • TensorFlow 2 is not directly supported, so models must be converted to ONNX first.
  • Building from source requires rustc 1.91.0 or newer.
  • The Metal and CUDA runtimes need Apple or NVIDIA GPUs respectively.

Topics

#neural-networks#onnx#inference#rust#wasm#edge-ai#machine-learning#artificial-intelligence#rust-library#tensorflow

FAQ

What is tract?
A self-contained Rust inference engine that loads ONNX and NNEF models and runs them on CPU, Apple Metal, NVIDIA CUDA, and WebAssembly. tract is a neural-network inference engine written in Rust, developed at Sonos. It loads ONNX and NNEF models, optimises them, and runs them on targets ranging from embedded ARM CPUs to NVIDIA and Apple GPUs, in the browser via WebAssembly, or on Linux, macOS and Windows machines. A Python package built on the same core is also published on PyPI.
Is tract free to use? What license does it have?
tract is open source under the Other license. Check the license terms before using it in commercial products.
Is tract still maintained?
Actively maintained. The last commit to tract was on Nov 6, 2026.
What can I use tract for?
Running wake-word or streaming speech recognition on embedded ARM devices. Shipping a small inference runtime without protobuf or training-framework loaders. Running ONNX or NNEF models in a Rust application on CPU or GPU. Executing image, speech or language models in the browser via WebAssembly.
How do I install tract?
Install the Python package from PyPI: pip install tract Load an ONNX model and run it with the Rust API, following the README quick start example
Where can I get tract?
The source code and installation instructions are on GitHub: https://github.com/sonos/tract

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects