Repolode
Submit
jesdga95

jesdga95/basalt

A local inference engine that runs the Qwen3.8-Flash-Next mixture-of-experts model on NVIDIA Blackwell GPUs, served through OpenAI- and Anthropic-compatible APIs.

View on GitHub
Stars
750
Forks
98
Open issues
71
Last commit
1 hour ago

What is it?

Basalt is a local inference engine built for the Qwen3.8-Flash-Next mixture-of-experts model. It targets Linux and Windows machines with an NVIDIA Blackwell GPU (compute capability 12.0) and CUDA 13.

Experts that do not fit in VRAM are kept in system RAM, and a second GPU can optionally serve as a peer expert tier. This lets a large MoE model run on a single workstation. The server exposes OpenAI- and Anthropic-compatible endpoints and includes a web UI for streaming replies.

Models are distributed as single `.basalt` files that bundle weights, tokenizer, MTP drafter, expert profile and vision projector. Pre-built packs in several quantization formats are published on Hugging Face, and configuration is handled through a JSON file and the `basalt` CLI. A `basalt-probe` tool measures the machine and suggests settings.

Key features

  • 01Keeps overflow experts in system RAM, with optional second-GPU expert tier.
  • 02Serves OpenAI- and Anthropic-compatible APIs from a local server.
  • 03Includes a web UI with streaming replies.
  • 04Packages weights, tokenizer, drafter and vision projector in one .basalt file.
  • 05Offers several quantized model packs, including NVFP4 and GGUF-derived formats.
  • 06Provides basalt-probe to measure hardware and suggest configuration values.

Use cases

  • Running a large MoE model on a single NVIDIA Blackwell workstation.
  • Serving a local OpenAI- or Anthropic-compatible endpoint for existing client tools.
  • Experimenting with quantization trade-offs for Qwen3.8-Flash-Next.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/jesdga95/basalt && cd basalt
  2. 2

    Build the project with CMake

    cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
  3. 3

    Install Python requirements and download the model

    pip install -r requirements.txt && hf download jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-NVFP4.basalt --local-dir models
  4. 4

    Start the server with your config file

    bin/basalt serve --config basalt_config.json

Things to know

  • Requires Linux or Windows with an NVIDIA Blackwell GPU (sm_120) and CUDA 13.
  • Needs enough system RAM to hold experts that do not fit in VRAM.
  • Model packs are under the Qwen Community License, not MIT.

In collections

Run LLMs locally

Topics

#llm-inference#mixture-of-experts#local-llm#cuda#nvidia-gpu#openai-compatible#quantization

FAQ

What is basalt?
A local inference engine that runs the Qwen3.8-Flash-Next mixture-of-experts model on NVIDIA Blackwell GPUs, served through OpenAI- and Anthropic-compatible APIs. Basalt is a local inference engine built for the Qwen3.8-Flash-Next mixture-of-experts model. It targets Linux and Windows machines with an NVIDIA Blackwell GPU (compute capability 12.0) and CUDA 13.
Is basalt free to use? What license does it have?
basalt is open source under the MIT license. Check the license terms before using it in commercial products.
Is basalt still maintained?
Actively maintained. The last commit to basalt was on Nov 7, 2026.
What can I use basalt for?
Running a large MoE model on a single NVIDIA Blackwell workstation. Serving a local OpenAI- or Anthropic-compatible endpoint for existing client tools. Experimenting with quantization trade-offs for Qwen3.8-Flash-Next.
How do I install basalt?
Clone the repository: git clone https://github.com/jesdga95/basalt && cd basalt Build the project with CMake: cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j Install Python requirements and download the model: pip install -r requirements.txt && hf download jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-NVFP4.basalt --local-dir models Start the server with your config file: bin/basalt serve --config basalt_config.json
Where can I get basalt?
The source code and installation instructions are on GitHub: https://github.com/jesdga95/basalt

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects