Repolode
Submit
mw00

mw00/project-maya

Self-hosted inference engine that runs large MoE models such as GLM-5.3-Flash on your own GPU, with OpenAI-compatible APIs.

Open-source alternative to:llama.cppollamavLLMExLlamaV3

View on GitHubWebsite
Stars
2,641
Forks
375
Open issues
154
Last commit
1 hour ago

What is it?

Project Maya is a local inference engine and server for running large mixture-of-experts models on hardware you already own rather than in a datacenter. Its engines target GLM-5.3-Flash (321B parameters, about 18B active per token) and DeepSeek-V4.1-Flash-1017 (748B). Experts used most stay in VRAM, the next ones in RAM, and the rest are read from an NVMe SSD; the engine measures GPUs, CPU, RAM and disk at startup and tunes itself to them.

It ships an engine, a server, a browser chat with a live monitor dashboard, and quantization tooling. Maya's own quants come in several sizes (Maya-S/S24/M/M-Derisked/L, 94.7-156.3 GB) and keep 97.7-99.2% of the FP8 model's zero-shot accuracy. Separate engines cover DeepSeek-V4.1-Flash-1017 (GGUF or EXL3 weights) and an experimental MiniMax-M3.1-Flash.

It runs on one to 16 NVIDIA GPUs (compute capability 7.0+), with AMD ROCm 7 and Windows marked experimental. The server exposes an OpenAI-compatible /v1 API and an Anthropic-compatible /v1/messages endpoint, streams tool calls, allows up to 8 parallel requests and four thinking levels, and includes --calibrate and --report for tuning and troubleshooting.

Key features

  • 01Tiered expert storage keeps hot experts in VRAM, the next ones in RAM and the rest on an NVMe SSD, moving them as the conversation changes.
  • 02Ships quants for GLM-5.3-Flash (Maya-S/S24/M/M-Derisked/L) plus separate engines for DeepSeek-V4.1-Flash-1017 and an experimental MiniMax-M3.1-Flash.
  • 03OpenAI-compatible /v1 API and Anthropic-compatible /v1/messages endpoint, so tools such as Claude Code can point at the local server.
  • 04/v1/responses supports text and image conversations, streaming, function tool calls, schema-validated output and cancellable background tasks.
  • 05Browser chat with per-chat instructions, edit and regenerate, HTML previews, and a live monitor of the model and hardware.
  • 06Runs on 1 to 16 NVIDIA GPUs (compute capability 7.0+); AMD ROCm 7 and Windows 10/11 are experimental.
  • 07Four thinking levels (Off, Low, High, Max) set globally or per request, plus parallel decoding of up to 8 requests at once.
  • 08KV streaming keeps attention caches in RAM with a VRAM window, so a 1M-token context fits on a single RTX 3090.
  • 09--calibrate tunes the engine to the machine in 10-15 minutes; --report writes a hardware and speed report for bug reports.

Use cases

  • Privately hosting a large MoE model on a workstation or small server where data must not leave the machine.
  • Pointing coding agents such as Claude Code at a local model through the Anthropic-compatible endpoint.
  • Sharing one machine between several developers or agent sessions using parallel requests.
  • Running long-document conversations at up to 1M tokens on a single consumer GPU with KV streaming.
  • Comparing quantization tiers by their reported accuracy retention and memory footprint before downloading.
  • Trying local large-model inference on AMD GPUs or a Windows workstation instead of a Linux CUDA box.

Quick start

  1. 1

    Clone the repository and open its folder

  2. 2

    Run the setup script and pick a model

    ./setup.sh --setup
  3. 3

    Start the server and chat in the browser

Things to know

  • Needs a powerful NVIDIA GPU setup, and AMD and Windows support is experimental.
  • Model files are very large, from about 94 GB to over 500 GB, and need fast local storage.
  • Some model options, such as Maya-M-Derisked and MiniMax-M3.1-Flash, are marked experimental.

In collections

Self-hosted appsRun LLMs locally

Topics

#llm#moe#llm-inference#local-llm#self-hosted#quantization#cuda#rocm#openai-api#deepseek

FAQ

What is project-maya?
Self-hosted inference engine that runs large MoE models such as GLM-5.3-Flash on your own GPU, with OpenAI-compatible APIs. Project Maya is a local inference engine and server for running large mixture-of-experts models on hardware you already own rather than in a datacenter. Its engines target GLM-5.3-Flash (321B parameters, about 18B active per token) and DeepSeek-V4.1-Flash-1017 (748B). Experts used most stay in VRAM, the next ones in RAM, and the rest are read from an NVMe SSD; the engine measures GPUs, CPU, RAM and disk at startup and tunes itself to them.
Is project-maya free to use? What license does it have?
project-maya is open source under the MIT license. Check the license terms before using it in commercial products.
Is project-maya still maintained?
Actively maintained. The last commit to project-maya was on Nov 7, 2026.
What can I use project-maya for?
Privately hosting a large MoE model on a workstation or small server where data must not leave the machine. Pointing coding agents such as Claude Code at a local model through the Anthropic-compatible endpoint. Sharing one machine between several developers or agent sessions using parallel requests. Running long-document conversations at up to 1M tokens on a single consumer GPU with KV streaming. Comparing quantization tiers by their reported accuracy retention and memory footprint before downloading. Trying local large-model inference on AMD GPUs or a Windows workstation instead of a Linux CUDA box.
How do I install project-maya?
Clone the repository and open its folder Run the setup script and pick a model: ./setup.sh --setup Start the server and chat in the browser
Where can I get project-maya?
The source code and installation instructions are on GitHub: https://github.com/mw00/project-maya

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects