Repolode
Submit
google

google/gemma.cpp

A lightweight, standalone C++ inference engine for running Google's Gemma models on CPU.

View on GitHub
Stars
7,079
Forks
673
Open issues
52
Last commit
22 hours ago

What is it?

gemma.cpp is a minimalist C++ inference engine for Gemma-2, Gemma-3 and PaliGemma-2 models. It is built for experimentation and research, aiming to sit between deployment-oriented C++ runtimes and Python-based ML frameworks, with a small core implementation that is easy to read, modify and embed in other projects.

The engine runs CPU-only inference using Google Highway for portable SIMD, and integrates weight compression directly into its matrix multiplication, including a custom fp8 format, bf16, f32 and non-uniform 4-bit quantization. It also includes a backward pass and Adam optimizer for Gemma research.

The project provides C++ APIs with streaming for single and batched queries, a basic interactive command-line app, and Python bindings via pybind11. Weights are downloaded from Kaggle, and builds use CMake or Bazel on Linux, Windows and macOS.

Key features

  • 01Runs Gemma-2, Gemma-3 and PaliGemma-2 inference on CPU.
  • 02Integrates weight compression such as fp8, bf16 and 4-bit NUQ into GEMM.
  • 03Uses Google Highway for portable SIMD and runtime ISA selection.
  • 04Offers C++ streaming APIs for single and batched inference.
  • 05Includes a basic interactive CLI and pybind11 Python bindings.
  • 06Supports a backward pass and Adam optimizer for research.

Use cases

  • Experimenting with low-level inference optimizations for Gemma models.
  • Embedding a small local LLM runtime into a C++ application.
  • Running Gemma or PaliGemma models on CPU-only machines for research.
  • Studying a compact, readable LLM implementation.

Quick start

  1. 1

    Install CMake and a Clang C++ compiler (C++17 or newer)

  2. 2

    Download model weights and tokenizer from Kaggle (gemma2-2b-it-sfp recommended) and extract the archive

    tar -xf archive.tar.gz
  3. 3

    Configure and build the gemma executable with CMake

    cmake --preset make
  4. 4

    Run gemma with the tokenizer and weights files

    ./gemma --tokenizer tokenizer.spm --weights gemma2-2b-it-sfp.sbs

Things to know

  • CPU-only inference; no GPU support.
  • Requires downloading model weights from Kaggle after accepting a consent form.
  • Building natively on Windows requires Visual Studio Build Tools with clang-cl; WSL is recommended instead.
  • Designed for experimentation and research rather than production deployment.

Topics

#llm#inference#cpp#gemma#cpu-inference#simd#quantization

FAQ

What is gemma.cpp?
A lightweight, standalone C++ inference engine for running Google's Gemma models on CPU. gemma.cpp is a minimalist C++ inference engine for Gemma-2, Gemma-3 and PaliGemma-2 models. It is built for experimentation and research, aiming to sit between deployment-oriented C++ runtimes and Python-based ML frameworks, with a small core implementation that is easy to read, modify and embed in other projects.
Is gemma.cpp free to use? What license does it have?
gemma.cpp is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is gemma.cpp still maintained?
Actively maintained. The last commit to gemma.cpp was on Nov 6, 2026.
What can I use gemma.cpp for?
Experimenting with low-level inference optimizations for Gemma models. Embedding a small local LLM runtime into a C++ application. Running Gemma or PaliGemma models on CPU-only machines for research. Studying a compact, readable LLM implementation.
How do I install gemma.cpp?
Install CMake and a Clang C++ compiler (C++17 or newer) Download model weights and tokenizer from Kaggle (gemma2-2b-it-sfp recommended) and extract the archive: tar -xf archive.tar.gz Configure and build the gemma executable with CMake: cmake --preset make Run gemma with the tokenizer and weights files: ./gemma --tokenizer tokenizer.spm --weights gemma2-2b-it-sfp.sbs
Where can I get gemma.cpp?
The source code and installation instructions are on GitHub: https://github.com/google/gemma.cpp

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects