Repolode
Submit
JakeATX

JakeATX/llamAmpere

A llama.cpp fork tuned for NVIDIA RTX 30-series Ampere cards, mainly the RTX 3090, for running Qwen3.8 models locally.

View on GitHub
Stars
227
Forks
31
Open issues
14
Last commit
20 hours ago

What is it?

llamAmpere is a fork of llama.cpp, the open-source program that runs language models on your own machine. It is tuned for NVIDIA RTX 30-series "Ampere" graphics cards, with the RTX 3090 and 3090 Ti as the main targets. The project focuses on Qwen3.8 models, including the 27B model with its built-in MTP draft head, which predicts several tokens ahead to speed up generation, and the Flash-Next variant.

The fork adds its own KV-cache quantization (SJ-KVaRN), multi-token drafting options, and expert offloading between GPU and system RAM. These are aimed at long-context work on a single consumer card, and the README describes setups for 12 GB and 24 GB GPUs with contexts of 200K tokens or more. The server exposes an OpenAI-compatible API on localhost.

Building requires CUDA 12 or newer, CMake, and a C++ compiler. Release notes in the repository list measured speeds and recommended settings for each card size.

Key features

  • 01Targets Ampere RTX 30-series cards, especially the RTX 3090 and 3090 Ti.
  • 02Runs Qwen3.8 27B with a built-in MTP draft head for faster generation.
  • 03Offers KV-cache quantization for contexts of 200K tokens or more.
  • 04Splits Flash-Next experts between GPU and system RAM.
  • 05Provides an OpenAI-compatible HTTP server through llama-server.

Use cases

  • Running a long-context Qwen3.8 model locally on a single RTX 3090.
  • Serving a local OpenAI-compatible endpoint for coding or RAG tools on a 24 GB card.
  • Trying 12 GB consumer GPUs with large context windows using the documented memory settings.

Quick start

  1. 1

    Clone llamAmpere at the v0.6.2 tag

    git clone -b v0.6.2 https://github.com/JakeATX/llamAmpere.git
  2. 2

    Enter the folder and configure the CUDA build for RTX 30-series (sm86)

    cd llamAmpere && cmake -S . -B build-sm86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DLLAMA_BUILD_BORINGSSL=ON
  3. 3

    Build the llama-server binary

    cmake --build build-sm86 -j8 --target llama-server
  4. 4

    Start the server with the Swift 1.5 27B model (downloads on first start; listens on http://127.0.0.1:8080)

    ./build-sm86/bin/llama-server -hf jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2 -c 204800 -ngl 99 -fa on -b 4096 -ub 1024 --parallel 1 --jinja --no-context-shift -ctk sjkvarn3 -ctv sjkvarn2 --sjkvarn-body-type auto --spec-type draft-mtp-adaptive --spec-draft-n-min-adaptive 3 --spec-draft-n-max 4 --spec-draft-p-min 0 --spec-mb-mtp

Things to know

  • Built and tuned for NVIDIA RTX 30-series (Ampere) cards, mainly the RTX 3090, so other GPUs are not its target.
  • The quick start needs a 24 GB card and the CUDA 12 or newer toolkit, and it builds from source.
  • Model files are very large, and the release notes say older file formats no longer load.

In collections

Run LLMs locally

Topics

#llm-inference#cuda#local-llm#quantization#long-context#speculative-decoding#qwen#gguf#ampere#consumer-gpu

FAQ

What is llamAmpere?
A llama.cpp fork tuned for NVIDIA RTX 30-series Ampere cards, mainly the RTX 3090, for running Qwen3.8 models locally. llamAmpere is a fork of llama.cpp, the open-source program that runs language models on your own machine. It is tuned for NVIDIA RTX 30-series "Ampere" graphics cards, with the RTX 3090 and 3090 Ti as the main targets. The project focuses on Qwen3.8 models, including the 27B model with its built-in MTP draft head, which predicts several tokens ahead to speed up generation, and the Flash-Next variant.
Is llamAmpere free to use? What license does it have?
llamAmpere is open source under the MIT license. Check the license terms before using it in commercial products.
Is llamAmpere still maintained?
Actively maintained. The last commit to llamAmpere was on Nov 6, 2026.
What can I use llamAmpere for?
Running a long-context Qwen3.8 model locally on a single RTX 3090. Serving a local OpenAI-compatible endpoint for coding or RAG tools on a 24 GB card. Trying 12 GB consumer GPUs with large context windows using the documented memory settings.
How do I install llamAmpere?
Clone llamAmpere at the v0.6.2 tag: git clone -b v0.6.2 https://github.com/JakeATX/llamAmpere.git Enter the folder and configure the CUDA build for RTX 30-series (sm86): cd llamAmpere && cmake -S . -B build-sm86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DLLAMA_BUILD_BORINGSSL=ON Build the llama-server binary: cmake --build build-sm86 -j8 --target llama-server Start the server with the Swift 1.5 27B model (downloads on first start; listens on http://127.0.0.1:8080): ./build-sm86/bin/llama-server -hf jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2 -c 204800 -ngl 99 -fa on -b 4096 -ub 1024 --parallel 1 --jinja --no-context-shift -ctk sjkvarn3 -ctv sjkvarn2 --sjkvarn-body-type auto --spec-type draft-mtp-adaptive --spec-draft-n-min-adaptive 3 --spec-draft-n-max 4 --spec-draft-p-min 0 --spec-mb-mtp
Where can I get llamAmpere?
The source code and installation instructions are on GitHub: https://github.com/JakeATX/llamAmpere

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects