Repolode
Submit
professorpalmer

professorpalmer/bonsai-ada-surgery

A launcher and patched llama.cpp setup that runs the 1.58-bit Ternary Bonsai 2 27B model with a 262k context on 12 GB and 8 GB NVIDIA GPUs.

View on GitHubWebsite
Stars
82
Forks
6
Open issues
4
Last commit
15 hours ago

What is it?

Ternary Bonsai 2 27B is a 1.58-bit ternary model distributed in the PTQ1_0 format at about 5.9 GB. This project serves it on consumer NVIDIA cards with the full 262,144-token context window, using a patched PrismML llama.cpp build and a launcher that sizes the K/V cache to the card: q4_0 on 12 GB and q3_0 on 8 GB.

The repository adds MTP drafting for speculative decoding, grammar-constrained tool calls, and an optional layer for agents that includes an OpenAI-compatible API and sandboxed Python execution. The model weights are unchanged. A Windows bundle with a PowerShell start script is provided, and Linux and source builds are documented separately.

The docs include benchmark logs for decode speed at different context depths, prefill throughput, and HumanEval results on an RTX 4070 and an RTX 2060 SUPER.

Key features

  • 01Runs the 262,144-token context window on 12 GB and 8 GB NVIDIA cards.
  • 02Sizes the K/V cache automatically per card, using q4_0 or q3_0.
  • 03Provides an OpenAI-compatible API on port 8080.
  • 04Adds MTP drafting for speculative decoding.
  • 05Supports grammar-constrained tool calls.
  • 06Ships a Windows bundle with a PowerShell launcher.

Use cases

  • Running a long-context local LLM on a 12 GB consumer GPU.
  • Serving a local OpenAI-compatible endpoint for coding or agent tools on an 8 GB RTX card.
  • Benchmarking ternary model decode and prefill speed across context depths.

Quick start

  1. 1

    Download the bundle zip from Releases and unzip it into the repo

  2. 2

    Put the GGUF model and MTP head file in the models folder

  3. 3

    Start the server; the OpenAI-compatible API runs on port 8080

    .\start-server.ps1

Things to know

  • The quick start is for Windows with NVIDIA GPUs; Linux needs a separate guide in docs/build.
  • Requires an NVIDIA card with 8 GB or more of VRAM to run the full window.
  • The license is listed as NOASSERTION, so the repo's license terms are unclear; the README says MIT for its own code.

Topics

#local-llm#quantization#llama-cpp#cuda#speculative-decoding#long-context#ternary#bonsai#qwen#rtx-4070

FAQ

What is bonsai-ada-surgery?
A launcher and patched llama.cpp setup that runs the 1.58-bit Ternary Bonsai 2 27B model with a 262k context on 12 GB and 8 GB NVIDIA GPUs. Ternary Bonsai 2 27B is a 1.58-bit ternary model distributed in the PTQ1_0 format at about 5.9 GB. This project serves it on consumer NVIDIA cards with the full 262,144-token context window, using a patched PrismML llama.cpp build and a launcher that sizes the K/V cache to the card: q4_0 on 12 GB and q3_0 on 8 GB.
Is bonsai-ada-surgery free to use? What license does it have?
bonsai-ada-surgery is open source under the Other license. Check the license terms before using it in commercial products.
Is bonsai-ada-surgery still maintained?
Actively maintained. The last commit to bonsai-ada-surgery was on Oct 20, 2026.
What can I use bonsai-ada-surgery for?
Running a long-context local LLM on a 12 GB consumer GPU. Serving a local OpenAI-compatible endpoint for coding or agent tools on an 8 GB RTX card. Benchmarking ternary model decode and prefill speed across context depths.
How do I install bonsai-ada-surgery?
Download the bundle zip from Releases and unzip it into the repo Put the GGUF model and MTP head file in the models folder Start the server; the OpenAI-compatible API runs on port 8080: .\start-server.ps1
Where can I get bonsai-ada-surgery?
The source code and installation instructions are on GitHub: https://github.com/professorpalmer/bonsai-ada-surgery

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects