Repolode
Submit
Luce-Org

Luce-Org/lucebox

A C++ LLM inference server for consumer GPUs and mixed AMD/NVIDIA machines, using speculative decoding, speculative prefill and custom kernels.

View on GitHubWebsite
Stars
2,954
Forks
286
Open issues
111
Last commit
16 hours ago

What is it?

Lucebox is an LLM inference server written in C++ that aims to make large local models faster on consumer hardware. It combines speculative decoding (a small drafter model proposes tokens that the target model verifies), speculative prefill for compressing long prompts, and kernels tuned per model and GPU. It exposes an OpenAI-compatible API on a local port, along with Anthropic Messages and Responses endpoints.

The project targets NVIDIA cards via CUDA and AMD cards via HIP/ROCm, and can split work across heterogeneous devices such as a Radeon R9700 paired with a Strix Halo APU, or spread experts onto SSD. Supported models include Qwen 3.x, Gemma 4, Laguna XS 2.1, and DeepSeek V4 variants, each with published GGUF weights and drafters. Docker images for CUDA 12+ and ROCm 6+ are provided.

The repository also includes client harness launchers for coding tools such as Claude Code, Codex, OpenCode and Open WebUI, plus image generation (MOSS-Image) and image input support. Benchmarks are reported per hardware setup and are not directly comparable across configurations.

Key features

  • 01Speculative decoding with drafter models for faster token generation.
  • 02Speculative prefill that compresses long prompts before the main model runs.
  • 03OpenAI-compatible, Anthropic Messages and Responses API endpoints.
  • 04Supports CUDA (NVIDIA) and HIP/ROCm (AMD) backends, including mixed-device placement.
  • 05Paged attention and continuous batching for serving several concurrent requests.
  • 06Prebuilt Docker images and launch profiles for specific hardware setups.

Use cases

  • Running large open-weight models locally on a single consumer GPU or AMD APU.
  • Serving a local OpenAI-compatible endpoint for coding agents and chat clients.
  • Handling long-context prompts on a home workstation with limited VRAM.
  • Experimenting with speculative decoding on NVIDIA or AMD hardware.

Things to know

  • The visible README gives no install or run commands, so setup steps are unclear from this page.
  • Benchmarks and profiles target specific hardware such as R9700, Strix Halo and RTX 3090, and the badges list CUDA 12+ or HIP 7+ requirements.

In collections

Run LLMs locallyRun and train AI on GPUs

Topics

#llm-inference#speculative-decoding#cuda#rocm#local-ai#gpu#c-plus-plus#cuda-kernels#dflash#heterogeneous-computing

FAQ

What is lucebox?
A C++ LLM inference server for consumer GPUs and mixed AMD/NVIDIA machines, using speculative decoding, speculative prefill and custom kernels. Lucebox is an LLM inference server written in C++ that aims to make large local models faster on consumer hardware. It combines speculative decoding (a small drafter model proposes tokens that the target model verifies), speculative prefill for compressing long prompts, and kernels tuned per model and GPU. It exposes an OpenAI-compatible API on a local port, along with Anthropic Messages and Responses endpoints.
Is lucebox free to use? What license does it have?
lucebox is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is lucebox still maintained?
Actively maintained. The last commit to lucebox was on Oct 24, 2026.
What can I use lucebox for?
Running large open-weight models locally on a single consumer GPU or AMD APU. Serving a local OpenAI-compatible endpoint for coding agents and chat clients. Handling long-context prompts on a home workstation with limited VRAM. Experimenting with speculative decoding on NVIDIA or AMD hardware.
Where can I get lucebox?
The source code and installation instructions are on GitHub: https://github.com/Luce-Org/lucebox

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects