Repolode
Submit
kindlingai

kindlingai/gr8

A from-scratch CUDA inference engine that runs the full GLM 5.3 model across three to six DGX Spark (GB10) boxes with an OpenAI-compatible API.

View on GitHub
Stars
62
Forks
13
Open issues
38
Last commit
3 hours ago

What is it?

gr8 is a local inference engine for the full GLM 5.3 model, a mixture-of-experts network with 256 experts, MLA attention with a sparse indexer, and one MTP layer. It is built from scratch in CUDA and runs tensor-parallel across three to six DGX Spark (GB10) machines, one process per box, with the ranks communicating over RoCE v2. At serve time it does not depend on vLLM, PyTorch or NCCL, and the head box exposes an OpenAI-compatible API on port 8100.

The engine handles concurrent requests by sharing decode steps across up to eight sessions, uses speculative decoding with an MTP layer or a DFlash2 drafter, and reuses KV prefixes so that a continued conversation only prefills its new tokens. Context-parallel KV spreads long contexts across ranks, and an experimental int4 KV format and remote KV nodes extend the usable context. Weights are fetched per box from a Hugging Face release, with several expert-width variants selectable at boot.

The project is in beta and targets GB10-class hardware specifically (sm_121a, 128 GB unified memory), with setup paths for bare metal, Docker Compose and Kubernetes.

Key features

  • 01Full GLM 5.3 inference with tensor parallelism across 3 to 6 GB10 boxes over RoCE v2.
  • 02OpenAI-compatible chat and completions API served from the head box.
  • 03Up to eight concurrent sessions share decode steps.
  • 04Speculative decoding with MTP or the DFlash2 drafter.
  • 05Prefix KV reuse so continued conversations prefill only new tokens.
  • 06Experimental int4 KV cache and remote KV nodes for longer contexts.

Use cases

  • Running a full-size GLM 5.3 model privately on a small cluster of DGX Spark or GB10 machines.
  • Serving an OpenAI-compatible endpoint to internal tools from local hardware.
  • Studying a from-scratch CUDA implementation of MoE and MLA inference.

Quick start

  1. 1

    Read the deployment guide and set the model and peer or mentat options

  2. 2

    Build the engine and front end

    ./gr8 build
  3. 3

    Start every box with the model

    ./gr8 glm53 up
  4. 4

    Run the smoke test against the head box's API

    ./gr8 smoke

Things to know

  • Requires three to six DGX Spark (GB10) boxes with 128 GB memory each; it will not run on ordinary hardware.
  • Beta release, so expect bugs and breaking changes.
  • Needs a RoCE v2 fabric or a ring cabling setup between the boxes, plus a LAN with specific TCP ports open.
  • Large model downloads and disk use of about 90 GB per box at TP=4.

In collections

Run LLMs locally

Topics

#llm-inference#cuda#glm#moe#tensor-parallelism#openai-compatible#speculative-decoding#local-llm

FAQ

What is gr8?
A from-scratch CUDA inference engine that runs the full GLM 5.3 model across three to six DGX Spark (GB10) boxes with an OpenAI-compatible API. gr8 is a local inference engine for the full GLM 5.3 model, a mixture-of-experts network with 256 experts, MLA attention with a sparse indexer, and one MTP layer. It is built from scratch in CUDA and runs tensor-parallel across three to six DGX Spark (GB10) machines, one process per box, with the ranks communicating over RoCE v2. At serve time it does not depend on vLLM, PyTorch or NCCL, and the head box exposes an OpenAI-compatible API on port 8100.
Is gr8 free to use? What license does it have?
gr8 is open source under the MIT license. Check the license terms before using it in commercial products.
Is gr8 still maintained?
Actively maintained. The last commit to gr8 was on Nov 7, 2026.
What can I use gr8 for?
Running a full-size GLM 5.3 model privately on a small cluster of DGX Spark or GB10 machines. Serving an OpenAI-compatible endpoint to internal tools from local hardware. Studying a from-scratch CUDA implementation of MoE and MLA inference.
How do I install gr8?
Read the deployment guide and set the model and peer or mentat options Build the engine and front end: ./gr8 build Start every box with the model: ./gr8 glm53 up Run the smoke test against the head box's API: ./gr8 smoke
Where can I get gr8?
The source code and installation instructions are on GitHub: https://github.com/kindlingai/gr8

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects