Repolode
Submit
trymirai

trymirai/uzu

A Rust inference engine for running AI models directly inside apps, with on-device execution on Apple devices and optional cloud models.

View on GitHubWebsite
Stars
2,459
Forks
148
Open issues
25
Last commit
4 hours ago

What is it?

uzu is an inference engine for running AI models inside your own application, aiming for zero network latency, data that stays on the device, and no per-request inference cost. It exposes a high-level API for chat sessions, with bindings for Rust, Python, Swift and TypeScript, and handles model downloading and inference configuration automatically.

The engine uses unified memory on Apple devices and is distributed as a Swift Package for iOS, macOS and Mac Catalyst, as well as npm and PyPI packages. Model configurations are unified, so adding support for a new model follows a common pattern, and computations are traceable for checking correctness against a reference implementation. Examples cover basic chat, streaming replies, cloud-hosted models, shared model instances, JSON-schema structured output and tool calls.

Each loaded model can use a significant amount of RAM, so the project recommends keeping one chat session loaded at a time. The documentation covers customizing each step of the pipeline.

Key features

  • 01Provides a high-level chat session API with bindings for Rust, Python, Swift and TypeScript.
  • 02Runs models on-device and uses unified memory on Apple devices.
  • 03Supports structured output constrained by a JSON schema.
  • 04Supports tool calls so models can invoke external functions.
  • 05Can reuse one loaded model across multiple chat sessions.
  • 06Also supports calling cloud models such as OpenAI's GPT-5.

Use cases

  • Adding an on-device LLM chat feature to an iOS or macOS app without sending user data to a server.
  • Generating JSON-formatted responses from a local model for downstream parsing.
  • Building an app that lets a model call local functions, such as fetching the device location.

Quick start

  1. 1

    Add the Python package with uv

    uv add uzu==0.6.11
  2. 2

    Create the engine, load the model, download it, and start a chat session using the Python example from the README

Things to know

  • Core inference is aimed at Apple devices; on-device execution is for iOS and macOS.
  • Cloud model options require an external service rather than fully local inference.
  • The Rust crate is pulled from a git branch rather than a stable crates.io release.

Topics

#inference#llm#rust#on-device-ai#apple-silicon#metal#swift#tts#ai#high-performance

FAQ

What is uzu?
A Rust inference engine for running AI models directly inside apps, with on-device execution on Apple devices and optional cloud models. uzu is an inference engine for running AI models inside your own application, aiming for zero network latency, data that stays on the device, and no per-request inference cost. It exposes a high-level API for chat sessions, with bindings for Rust, Python, Swift and TypeScript, and handles model downloading and inference configuration automatically.
Is uzu free to use? What license does it have?
uzu is open source under the MIT license. Check the license terms before using it in commercial products.
Is uzu still maintained?
Actively maintained. The last commit to uzu was on Oct 21, 2026.
What can I use uzu for?
Adding an on-device LLM chat feature to an iOS or macOS app without sending user data to a server. Generating JSON-formatted responses from a local model for downstream parsing. Building an app that lets a model call local functions, such as fetching the device location.
How do I install uzu?
Add the Python package with uv: uv add uzu==0.6.11 Create the engine, load the model, download it, and start a chat session using the Python example from the README
Where can I get uzu?
The source code and installation instructions are on GitHub: https://github.com/trymirai/uzu

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects