Repolode
Submit
midagedev

midagedev/bloomery

A Rust LLM inference server for mixture-of-experts models larger than the GPU, with a llama-server-compatible HTTP API, on NVIDIA GPUs and Apple Silicon.

View on GitHub
Stars
60
Forks
7
Open issues
49
Last commit
19 hours ago

What is it?

bloomery serves large mixture-of-experts language models on a single machine by splitting the routed experts between GPU memory, host RAM and, when needed, NVMe storage. The engine tracks which experts the decoder calls most and moves them onto the card while it runs. It targets NVIDIA cards, including the DGX Spark (GB10), and Apple Silicon through a Metal backend.

The server exposes the same HTTP API as llama-server, with OpenAI and Anthropic routes, GGUF input and llama's placement flags such as -ngl, --cpu-moe and -ot, so existing clients work unchanged. It also includes a prompt cache kept in host RAM, batched requests across multiple slots, and speculative decoding where a model provides a draft. Models can be loaded from GGUF files, or from NVFP4 and MLX checkpoint folders in some cases.

The CUDA kernels are written in Rust and compiled with cuda-oxide. The project is MIT licensed and ships prebuilt releases through an installer, Homebrew, Docker and tarballs.

Key features

  • 01Runs MoE models whose weights exceed GPU memory by placing experts across cards, host RAM and NVMe.
  • 02Exposes an llama-server-compatible HTTP API with OpenAI and Anthropic routes.
  • 03Reads GGUF files and llama's placement flags, so existing clients work unchanged.
  • 04Keeps saved prompt states in host RAM so requests sharing a prefix can reuse them.
  • 05Supports multiple concurrent slots and speculative decoding where a draft is available.
  • 06Includes a Metal backend for Apple Silicon Macs.

Use cases

  • Running a large MoE model such as Qwen3-30B-A3B or GLM on a single workstation with one 24 GB GPU.
  • Serving local models on an NVIDIA DGX Spark or RTX 5090 for coding agents.
  • Swapping a llama-server deployment for a Rust-based server with the same client interface.
  • Running mixture-of-experts models on an Apple Silicon Mac with a Metal GPU.

Quick start

  1. 1

    Run the official install script from the README

  2. 2

    Start the server with a model from Hugging Face

    ~/.local/bin/bloomery-serve --hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M
  3. 3

    Send a test chat request to the server

    curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"messages": [{"role": "user", "content": "Hello"}], "max_tokens": 64}'

Things to know

  • Requires an NVIDIA GPU with compute capability 8.6 or newer, or Apple Silicon on macOS 26.2 or newer.
  • Apple Silicon support is limited to specific architectures and mostly unverified, with one slot and text only.
  • Linux x86-64 needs an NVIDIA driver of R580 or newer, and Windows runs only through WSL2.
  • The project is young, with 57 stars and version 0.3.x.

In collections

Run LLMs locally

Topics

#llm#inference#moe#cuda#gguf#llama-cpp#rust#metal#deepseek#glm

FAQ

What is bloomery?
A Rust LLM inference server for mixture-of-experts models larger than the GPU, with a llama-server-compatible HTTP API, on NVIDIA GPUs and Apple Silicon. bloomery serves large mixture-of-experts language models on a single machine by splitting the routed experts between GPU memory, host RAM and, when needed, NVMe storage. The engine tracks which experts the decoder calls most and moves them onto the card while it runs. It targets NVIDIA cards, including the DGX Spark (GB10), and Apple Silicon through a Metal backend.
Is bloomery free to use? What license does it have?
bloomery is open source under the MIT license. Check the license terms before using it in commercial products.
Is bloomery still maintained?
Actively maintained. The last commit to bloomery was on Nov 6, 2026.
What can I use bloomery for?
Running a large MoE model such as Qwen3-30B-A3B or GLM on a single workstation with one 24 GB GPU. Serving local models on an NVIDIA DGX Spark or RTX 5090 for coding agents. Swapping a llama-server deployment for a Rust-based server with the same client interface. Running mixture-of-experts models on an Apple Silicon Mac with a Metal GPU.
How do I install bloomery?
Run the official install script from the README Start the server with a model from Hugging Face: ~/.local/bin/bloomery-serve --hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M Send a test chat request to the server: curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"messages": [{"role": "user", "content": "Hello"}], "max_tokens": 64}'
Where can I get bloomery?
The source code and installation instructions are on GitHub: https://github.com/midagedev/bloomery

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects