Repolode
Submit
ggml-org

ggml-org/llama.cpp

LLM inference engine in plain C/C++ for running large language and vision-language models locally or in the cloud across many hardware backends.

Open-source alternative to:OpenAI APIOllama

View on GitHubWebsite
Stars
131,780
Forks
24,581
Open issues
2,565
Last commit
6 hours ago

What is it?

llama.cpp is an inference engine for large language models (LLMs) and vision-language models (VLMs), written in plain C/C++ with no external dependencies. It aims to make model inference easy to set up and fast across a wide range of hardware, both on personal machines and in the cloud. It is built on the ggml tensor library.

The project covers a broad set of hardware: Apple silicon through ARM NEON, Accelerate and Metal, x86 CPUs with AVX/AVX2/AVX512/AMX, RISC-V vector extensions, and GPUs from NVIDIA (CUDA), AMD (HIP), Moore Threads (MUSA), Intel (SYCL), and others via Vulkan, OpenCL and WebGPU. It supports integer quantization from 1.5-bit up to 8-bit to reduce memory use, and can split models between CPU and GPU when they exceed VRAM.

The repository ships command-line tools, including `llama cli` for interactive sessions and `llama serve` for an OpenAI-compatible API server with a built-in web UI. Models can be pulled directly from Hugging Face, and Docker images and prebuilt binaries are available.

Key features

  • 01Plain C/C++ implementation with no external dependencies
  • 02Hardware acceleration for Apple Metal, x86 AVX/AMX, CUDA, HIP, Vulkan, SYCL and more
  • 03Integer quantization from 1.5-bit to 8-bit to reduce memory use
  • 04CPU+GPU hybrid inference for models larger than available VRAM
  • 05OpenAI-compatible API server with a built-in web UI

Use cases

  • Running open-weight LLMs locally on a laptop or desktop without a cloud service
  • Serving a local OpenAI-compatible endpoint for applications that already use that API
  • Running quantized models on limited-memory or edge hardware
  • Experimenting with inference on specific accelerators such as Apple silicon, Vulkan GPUs or Ascend NPUs

Quick start

  1. 1

    Run the official install script from the README

  2. 2

    Download pre-built binaries from the releases page

  3. 3

    Download and run a model directly from Hugging Face

    llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
  4. 4

    Launch the OpenAI-compatible API server

    llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Things to know

  • Some backends, such as CUDA, Metal, and SYCL, only apply to specific hardware.
  • The OpenVINO backend is still marked as in progress.

Topics

#llm#inference#gguf#quantization#cpp#local-llm#gpu#openai-api#ggml

FAQ

What is llama.cpp?
LLM inference engine in plain C/C++ for running large language and vision-language models locally or in the cloud across many hardware backends. llama.cpp is an inference engine for large language models (LLMs) and vision-language models (VLMs), written in plain C/C++ with no external dependencies. It aims to make model inference easy to set up and fast across a wide range of hardware, both on personal machines and in the cloud. It is built on the ggml tensor library.
Is llama.cpp free to use? What license does it have?
llama.cpp is open source under the MIT license. Check the license terms before using it in commercial products.
Is llama.cpp still maintained?
Actively maintained. The last commit to llama.cpp was on Oct 20, 2026.
What can I use llama.cpp for?
Running open-weight LLMs locally on a laptop or desktop without a cloud service Serving a local OpenAI-compatible endpoint for applications that already use that API Running quantized models on limited-memory or edge hardware Experimenting with inference on specific accelerators such as Apple silicon, Vulkan GPUs or Ascend NPUs
How do I install llama.cpp?
Run the official install script from the README Download pre-built binaries from the releases page Download and run a model directly from Hugging Face: llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF Launch the OpenAI-compatible API server: llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
Where can I get llama.cpp?
The source code and installation instructions are on GitHub: https://github.com/ggml-org/llama.cpp

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects