Repolode
Submit
EricLBuehler

EricLBuehler/candle-vllm

A Rust-based platform for running and serving local LLMs, with an OpenAI-compatible API server and built-in Web UI.

View on GitHub
Stars
740
Forks
91
Open issues
31
Last commit
20 days ago

What is it?

Candle-vLLM is an inference and serving platform for local large language models, written in Rust on top of the Candle framework. It exposes an OpenAI-compatible API server, so existing clients that speak the OpenAI protocol can point at a locally hosted model. An optional ChatGPT-style Web UI can be launched alongside the API.

The project targets efficient serving with features such as continuous batching, PagedAttention-style KV cache management, prefix caching, CUDA Graphs, and chunked prefill. It supports CUDA on Linux and Metal on macOS, loads models from HuggingFace IDs, local safetensors directories, or GGUF files, and offers in-situ quantization as well as GPTQ, AWQ, Marlin, FP8, and MXFP4/NVFP4 formats. Multi-GPU tensor parallelism and TCP-based multi-node inference are also supported, along with MCP and OpenAI-style tool calling.

Key features

  • 01OpenAI-compatible API server with streaming responses
  • 02Continuous batching, prefix caching and CUDA Graph support
  • 03TurboQuant and FP8 KV cache options for longer context
  • 04Multi-GPU tensor parallelism and TCP-based multi-node inference
  • 05Supports GGUF, safetensors, GPTQ, AWQ, Marlin and FP8/FP4 model formats
  • 06Built-in ChatGPT-style Web UI and MCP tool calling

Use cases

  • Serving an open-weight LLM on your own GPU server behind an OpenAI-compatible endpoint
  • Running quantized GGUF models on an Apple Silicon Mac with Metal acceleration
  • Spreading a large model across several GPUs or multiple machines
  • Connecting coding agents or other OpenAI-API clients to a self-hosted model

Quick start

  1. 1

    Install with the one-line install script (DEB or binary)

  2. 2

    Or build from source with cargo (CUDA build shown)

    cargo install --features cuda,nccl,flashinfer,cutlass --path .
  3. 3

    Run a HuggingFace model with the built-in Web UI

    candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server

Things to know

  • Supports CUDA (Linux) and Metal (macOS) only.
  • Multi-GPU runs need the number of GPUs to be a power of 2.
  • Several features, such as FP8 models and some kernels, depend on specific NVIDIA GPU architectures (SM70+ or SM90+).

In collections

Run LLMs locally

Topics

#llm-inference#rust#openai-compatible#local-llm#cuda#metal#quantization#model-serving

FAQ

What is candle-vllm?
A Rust-based platform for running and serving local LLMs, with an OpenAI-compatible API server and built-in Web UI. Candle-vLLM is an inference and serving platform for local large language models, written in Rust on top of the Candle framework. It exposes an OpenAI-compatible API server, so existing clients that speak the OpenAI protocol can point at a locally hosted model. An optional ChatGPT-style Web UI can be launched alongside the API.
Is candle-vllm free to use? What license does it have?
candle-vllm is open source under the MIT license. Check the license terms before using it in commercial products.
Is candle-vllm still maintained?
Actively maintained. The last commit to candle-vllm was on Oct 18, 2026.
What can I use candle-vllm for?
Serving an open-weight LLM on your own GPU server behind an OpenAI-compatible endpoint Running quantized GGUF models on an Apple Silicon Mac with Metal acceleration Spreading a large model across several GPUs or multiple machines Connecting coding agents or other OpenAI-API clients to a self-hosted model
How do I install candle-vllm?
Install with the one-line install script (DEB or binary) Or build from source with cargo (CUDA build shown): cargo install --features cuda,nccl,flashinfer,cutlass --path . Run a HuggingFace model with the built-in Web UI: candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server
Where can I get candle-vllm?
The source code and installation instructions are on GitHub: https://github.com/EricLBuehler/candle-vllm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects