Repolode
Submit
utilityai

utilityai/llama-cpp-rs

Rust bindings for llama.cpp, letting Rust programs load and run GGUF language models locally through the llama-cpp-2 crate.

View on GitHub
Stars
664
Forks
261
Open issues
62
Last commit
2 days ago

What is it?

llama-cpp-rs provides Rust bindings to llama.cpp, the C/C++ library for running large language models. It is published as the llama-cpp-2 crate on crates.io, with the llama-cpp-sys-2 crate containing the low-level bindings.

The project is designed to track llama.cpp closely and stay close to the raw C API rather than wrapping it in a heavy abstraction. As a result, it does not follow semver strictly, and its versions are updated to match upstream llama.cpp changes.

The repository includes a simple example that loads a GGUF model, such as a quantized Llama 2 model downloaded from Hugging Face, and generates text. CUDA GPU support is enabled through a cargo feature flag.

Key features

  • 01Provides Rust bindings for llama.cpp via the llama-cpp-2 crate.
  • 02Loads GGUF model files for local text generation.
  • 03Includes low-level llama-cpp-sys-2 bindings alongside the higher-level crate.
  • 04Supports CUDA GPU acceleration through a cargo feature flag.
  • 05Stays close to the upstream llama.cpp API for quick updates.

Use cases

  • Embedding local LLM inference into a Rust application without a separate server.
  • Running quantized GGUF models such as Llama 2 on a local machine.
  • Building Rust tooling that needs direct access to llama.cpp functionality.

Quick start

  1. 1

    Clone the repository with its submodules

    git clone --recursive https://github.com/utilityai/llama-cpp-rs
  2. 2

    Change into the project directory

    cd llama-cpp-rs
  3. 3

    Run the simple example (add --features cuda if you have a CUDA GPU)

    cargo run --release --bin simple -- --prompt "The way to kill a linux process is" hf-model TheBloke/Llama-2-7B-GGUF llama-2-7b.Q4_K_M.gguf

Things to know

  • The project is built to track llama.cpp closely, so it is close to raw bindings and does not follow semver meaningfully.
  • Building the example requires cloning git submodules and a working Rust toolchain.

In collections

Run LLMs locally

Topics

#rust#llm#llama-cpp#inference#gguf#bindings#local-llm

FAQ

What is llama-cpp-rs?
Rust bindings for llama.cpp, letting Rust programs load and run GGUF language models locally through the llama-cpp-2 crate. llama-cpp-rs provides Rust bindings to llama.cpp, the C/C++ library for running large language models. It is published as the llama-cpp-2 crate on crates.io, with the llama-cpp-sys-2 crate containing the low-level bindings.
Is llama-cpp-rs free to use? What license does it have?
llama-cpp-rs is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is llama-cpp-rs still maintained?
Actively maintained. The last commit to llama-cpp-rs was on Nov 4, 2026.
What can I use llama-cpp-rs for?
Embedding local LLM inference into a Rust application without a separate server. Running quantized GGUF models such as Llama 2 on a local machine. Building Rust tooling that needs direct access to llama.cpp functionality.
How do I install llama-cpp-rs?
Clone the repository with its submodules: git clone --recursive https://github.com/utilityai/llama-cpp-rs Change into the project directory: cd llama-cpp-rs Run the simple example (add --features cuda if you have a CUDA GPU): cargo run --release --bin simple -- --prompt "The way to kill a linux process is" hf-model TheBloke/Llama-2-7B-GGUF llama-2-7b.Q4_K_M.gguf
Where can I get llama-cpp-rs?
The source code and installation instructions are on GitHub: https://github.com/utilityai/llama-cpp-rs

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects