Repolode
Submit
ikawrakow

ikawrakow/ik_llama.cpp

A llama.cpp fork focused on CPU inference speed, with extra quantization types and features for running GGUF LLMs locally.

View on GitHub
Stars
3,314
Forks
494
Open issues
113
Last commit
yesterday

What is it?

ik_llama.cpp is a fork of llama.cpp, started in June 2024, that runs large language models locally from GGUF files. It adds additional quantization types (such as the IQK and Trellis families) and aims for better CPU performance, with the most complete support on AVX2 or newer x86 CPUs and ARM NEON, and on NVIDIA GPUs with CUDA (Turing or newer).

Beyond the core inference engine, the fork includes features such as MLA and FlashMLA attention for DeepSeek models, tensor overrides for deciding which weights live on GPU or CPU, a graph split mode for multi-GPU setups, speculative decoding variants, and an OpenAI-compatible llama-server with a built-in web UI. Many model architectures are supported, including recent Qwen, GLM, DeepSeek and Gemma releases.

The project notes that ROCm, Vulkan, Metal and older GPUs are not actively maintained, so users on those backends should check the documentation before relying on it.

Key features

  • 01Additional quantization types such as IQK and Trellis quants for smaller, faster CPU inference.
  • 02MLA and FlashMLA attention implementations for DeepSeek models.
  • 03Tensor overrides to control which weights are stored on GPU or CPU.
  • 04Graph split mode for multi-GPU setups.
  • 05OpenAI-compatible llama-server with a built-in web UI and function calling support.
  • 06Speculative decoding options, including MTP and n-gram based methods.

Use cases

  • Running quantized GGUF models on a CPU-heavy or hybrid CPU/GPU machine.
  • Serving local LLM endpoints with an OpenAI-compatible API.
  • Running large MoE models such as DeepSeek on a workstation with limited VRAM.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/ikawrakow/ik_llama.cpp
  2. 2

    Build for CPU with CMake

    cmake -B build -DGGML_NATIVE=ON
  3. 3

    Compile the build in Release mode

    cmake --build build --config Release -j$(nproc)
  4. 4

    Download a .gguf model file and start the server

    ./build/bin/llama-server --model /my_local_files/gguf/Qwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096

Things to know

  • Only CPU (AVX2 or better, ARM NEON) and CUDA (Turing or newer) backends are fully supported.
  • ROCm, Vulkan, Metal, old Nvidia GPUs and AVX-only CPUs are not supported by maintainers.
  • Some Unsloth _XL models with f16 tensors may not work.
  • Mainline llama.cpp model and quant compatibility may differ from this fork.

Topics

#llm#inference#quantization#cpp#gguf#cpu-inference#llama

FAQ

What is ik_llama.cpp?
A llama.cpp fork focused on CPU inference speed, with extra quantization types and features for running GGUF LLMs locally. ik_llama.cpp is a fork of llama.cpp, started in June 2024, that runs large language models locally from GGUF files. It adds additional quantization types (such as the IQK and Trellis families) and aims for better CPU performance, with the most complete support on AVX2 or newer x86 CPUs and ARM NEON, and on NVIDIA GPUs with CUDA (Turing or newer).
Is ik_llama.cpp free to use? What license does it have?
ik_llama.cpp is open source under the MIT license. Check the license terms before using it in commercial products.
Is ik_llama.cpp still maintained?
Actively maintained. The last commit to ik_llama.cpp was on Oct 20, 2026.
What can I use ik_llama.cpp for?
Running quantized GGUF models on a CPU-heavy or hybrid CPU/GPU machine. Serving local LLM endpoints with an OpenAI-compatible API. Running large MoE models such as DeepSeek on a workstation with limited VRAM.
How do I install ik_llama.cpp?
Clone the repository: git clone https://github.com/ikawrakow/ik_llama.cpp Build for CPU with CMake: cmake -B build -DGGML_NATIVE=ON Compile the build in Release mode: cmake --build build --config Release -j$(nproc) Download a .gguf model file and start the server: ./build/bin/llama-server --model /my_local_files/gguf/Qwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096
Where can I get ik_llama.cpp?
The source code and installation instructions are on GitHub: https://github.com/ikawrakow/ik_llama.cpp

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects