Repolode
Submit
professorpalmer

professorpalmer/mirai-s-ada

Serving setup for Mirai S 27B (a Qwen3.8-27B GGUF) on a 12 GB RTX 4070, with a full 262k-token window and q8_0 K/V cache.

View on GitHub
Stars
101
Forks
10
Open issues
0
Last commit
15 hours ago

What is it?

This repository is a local serving setup for Mirai S 27B, a Qwen3.8-27B-based model, running on a single consumer GPU with 12 GB of VRAM. It uses a GGUF conversion of the model with unchanged weights and provides a launcher for Windows and a `start-server.sh` script for Linux.

The setup aims to keep the full 262,144-token context window usable on a 12 GB card by using a q8_0 K/V cache and a tiered cache scheme. It also enables MTP drafting across context depths. The project publishes decode speed measurements at several context lengths, along with HumanEval and AppWorld results.

An optional agents layer is included for running the model inside agent workflows, and the repository documents build steps, settings, known limits and saved benchmark data.

Key features

  • 01Runs a 262,144-token context window on a 12 GB RTX 4070 with a q8_0 K/V cache.
  • 02Uses MTP drafting at every context depth.
  • 03Provides a Windows launcher and a Linux start script.
  • 04Includes an optional agents layer for agent workflows.
  • 05Publishes decode speed and benchmark results at multiple context lengths.

Use cases

  • Running a large local LLM with long context on a single 12 GB consumer GPU.
  • Experimenting with long-context agent workflows on local hardware.
  • Reproducing the published speed and quality measurements on similar hardware.

Quick start

  1. 1

    Read the setup guide for Windows and Linux

  2. 2

    Run the Linux server script

    start-server.sh

Things to know

  • Targets a 12 GB RTX 4070 specifically; other hardware is not covered by the README's measurements.
  • Uses the NOASSERTION license, so the licensing terms are unclear and need checking before use.
  • Relies on a third-party GGUF conversion and an experimental upstream model.

Topics

#llm#gguf#local-inference#quantization#long-context#llama-cpp#gpu

FAQ

What is mirai-s-ada?
Serving setup for Mirai S 27B (a Qwen3.8-27B GGUF) on a 12 GB RTX 4070, with a full 262k-token window and q8_0 K/V cache. This repository is a local serving setup for Mirai S 27B, a Qwen3.8-27B-based model, running on a single consumer GPU with 12 GB of VRAM. It uses a GGUF conversion of the model with unchanged weights and provides a launcher for Windows and a `start-server.sh` script for Linux.
Is mirai-s-ada free to use? What license does it have?
mirai-s-ada is open source under the Other license. Check the license terms before using it in commercial products.
Is mirai-s-ada still maintained?
Actively maintained. The last commit to mirai-s-ada was on Oct 20, 2026.
What can I use mirai-s-ada for?
Running a large local LLM with long context on a single 12 GB consumer GPU. Experimenting with long-context agent workflows on local hardware. Reproducing the published speed and quality measurements on similar hardware.
How do I install mirai-s-ada?
Read the setup guide for Windows and Linux Run the Linux server script: start-server.sh
Where can I get mirai-s-ada?
The source code and installation instructions are on GitHub: https://github.com/professorpalmer/mirai-s-ada

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects