Repolode
Submit
huggingface

huggingface/candle

A minimalist ML framework for Rust with GPU support, designed for fast, lightweight inference and deployment without Python.

Open-source alternative to:PyTorch

View on GitHub
Stars
21,279
Forks
1,855
Open issues
964
Last commit
5 days ago

What is it?

Candle is a machine learning framework written in Rust, with an API that feels similar to PyTorch. It provides tensors, devices and common ops, with an optimized CPU backend (optional MKL on x86, Accelerate on macOS), a CUDA backend with multi-GPU distribution via NCCL, and WASM support for running models in the browser.

The project aims to make serverless inference practical by producing small binaries instead of pulling in a full Python ML stack, so Python can be removed from production workloads. It loads models from safetensors, npz, ggml and PyTorch files, and supports quantization using llama.cpp quantized types.

The repository includes a model zoo covering large language models such as LLaMA, Mistral, Mixtral, Qwen and Gemma, plus Stable Diffusion, Whisper, YOLO, SAM and CLIP, along with command-line examples for each.

Key features

  • 01PyTorch-like API for tensors, ops and model code in Rust.
  • 02CPU backend with optional MKL (x86) and Accelerate (macOS), plus a CUDA backend.
  • 03Multi-GPU distribution via NCCL.
  • 04WASM support for running models in a browser.
  • 05Loads safetensors, npz, ggml and PyTorch model files.
  • 06Quantization using llama.cpp quantized types.

Use cases

  • Deploying model inference as small serverless binaries without a Python runtime.
  • Running LLMs, speech recognition or image models inside Rust services.
  • Running models client-side in the browser via WASM.
  • Porting PyTorch models to a Rust codebase.

Quick start

  1. 1

    Add candle-core to your Rust project and run the matrix multiplication example

    cargo run
  2. 2

    Enable CUDA support by switching the device to GPU in your code

  3. 3

    Run a command line example with CUDA enabled

    cargo run --example quantized --release

Things to know

  • Requires a Rust toolchain and, for GPU use, a separate CUDA setup.
  • Many model examples need model weight files downloaded separately.

Topics

#machine-learning#rust#inference#deep-learning#cuda#wasm#llm#quantization

FAQ

What is candle?
A minimalist ML framework for Rust with GPU support, designed for fast, lightweight inference and deployment without Python. Candle is a machine learning framework written in Rust, with an API that feels similar to PyTorch. It provides tensors, devices and common ops, with an optimized CPU backend (optional MKL on x86, Accelerate on macOS), a CUDA backend with multi-GPU distribution via NCCL, and WASM support for running models in the browser.
Is candle free to use? What license does it have?
candle is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is candle still maintained?
Actively maintained. The last commit to candle was on Nov 2, 2026.
What can I use candle for?
Deploying model inference as small serverless binaries without a Python runtime. Running LLMs, speech recognition or image models inside Rust services. Running models client-side in the browser via WASM. Porting PyTorch models to a Rust codebase.
How do I install candle?
Add candle-core to your Rust project and run the matrix multiplication example: cargo run Enable CUDA support by switching the device to GPU in your code Run a command line example with CUDA enabled: cargo run --example quantized --release
Where can I get candle?
The source code and installation instructions are on GitHub: https://github.com/huggingface/candle

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects