Repolode
Submit
EricLBuehler

EricLBuehler/mistral.rs

Rust-based LLM inference engine serving OpenAI- and Anthropic-compatible APIs, with quantization, multimodal models, and agent tooling.

Open-source alternative to:OpenAI API

View on GitHub
Stars
7,761
Forks
742
Open issues
415
Last commit
yesterday

What is it?

mistral.rs is an LLM inference engine written in Rust. It loads Hugging Face models and GGUF files, detects their architecture and chat template automatically, and runs them on CUDA, Metal, or CPU. A single `mistralrs serve` process exposes OpenAI-compatible `/v1` endpoints and Anthropic-compatible Messages endpoints, and includes a built-in web UI at `/ui`.

The project covers several areas that are often split across tools: quantization (ISQ, GGUF, GPTQ, AWQ, FP8 and others), PagedAttention and prefix caching for batched serving, LoRA and X-LoRA adapters, and multi-GPU inference. It also handles text, vision, video, and audio input, plus speech and image generation and embeddings, across a broad list of model architectures.

Beyond serving, it includes an agentic loop with tool calling, sandboxed Python and shell execution, web search, and MCP client support. Python and Rust SDKs let you run the engine in-process without starting a server.

Key features

  • 01Serves OpenAI-compatible and Anthropic-compatible APIs from one process.
  • 02Loads Hugging Face models and GGUF files with automatic architecture and template detection.
  • 03Supports in-situ quantization and GGUF, GPTQ, AWQ, FP8 and other formats.
  • 04Uses PagedAttention and prefix caching for continuous batching on CUDA and Apple Silicon.
  • 05Includes LoRA and X-LoRA adapter support and multi-GPU inference.
  • 06Provides tool calling, Python and shell execution, web search, and MCP client integration.

Use cases

  • Running open-weight models locally behind an OpenAI- or Anthropic-compatible endpoint.
  • Embedding LLM inference into a Rust or Python application without a separate server.
  • Serving quantized models on limited GPU memory or Apple Silicon.
  • Building agents that run code, shell commands, or web searches with tool calls.

Quick start

  1. 1

    Run the official install script for your platform from the README

  2. 2

    Start a model and its server

    mistralrs serve -m Qwen/Qwen3-4B --quant 4
  3. 3

    Open the web UI in your browser and send a message

Things to know

  • Installation relies on a remote install script rather than a standard package-manager install.
  • Accelerator-specific builds (CUDA, Metal, MKL, Accelerate) need separate setup per the getting-started guide.

Topics

#llm#inference#rust#quantization#openai-compatible#gguf#cuda#agents#uqff

FAQ

What is mistral.rs?
Rust-based LLM inference engine serving OpenAI- and Anthropic-compatible APIs, with quantization, multimodal models, and agent tooling. mistral.rs is an LLM inference engine written in Rust. It loads Hugging Face models and GGUF files, detects their architecture and chat template automatically, and runs them on CUDA, Metal, or CPU. A single `mistralrs serve` process exposes OpenAI-compatible `/v1` endpoints and Anthropic-compatible Messages endpoints, and includes a built-in web UI at `/ui`.
Is mistral.rs free to use? What license does it have?
mistral.rs is open source under the MIT license. Check the license terms before using it in commercial products.
Is mistral.rs still maintained?
Actively maintained. The last commit to mistral.rs was on Oct 20, 2026.
What can I use mistral.rs for?
Running open-weight models locally behind an OpenAI- or Anthropic-compatible endpoint. Embedding LLM inference into a Rust or Python application without a separate server. Serving quantized models on limited GPU memory or Apple Silicon. Building agents that run code, shell commands, or web searches with tool calls.
How do I install mistral.rs?
Run the official install script for your platform from the README Start a model and its server: mistralrs serve -m Qwen/Qwen3-4B --quant 4 Open the web UI in your browser and send a message
Where can I get mistral.rs?
The source code and installation instructions are on GitHub: https://github.com/EricLBuehler/mistral.rs

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects