Repolode
Submit
Niko1221

Niko1221/Strata

Local inference engine that runs the large Qwen3.8-Flash-Next model on consumer NVIDIA or AMD GPUs under Windows or Linux, with OpenAI and Anthropic compatible APIs.

View on GitHub
Stars
27,806
Forks
2,945
Open issues
1,362
Last commit
2 days ago

What is it?

Strata is a local inference engine that runs Qwen3.8-Flash-Next, a very large model normally served from datacenter hardware, on an ordinary gaming PC. It requires an NVIDIA or AMD graphics card with at least 12 GB of VRAM, 32 GB or more of RAM, and Windows 10/11 or Linux. An installer detects the hardware, recommends a model size that fits the available memory, downloads the model and starts a local server.

The model is split into many experts. Strata keeps the most frequently used experts on the GPU, holds all of them in system RAM, and reads the rest from an SSD lookup table, so the model fits in consumer memory. It also uses a small helper model to draft several tokens that the large model verifies at once. The server exposes OpenAI-compatible endpoints, an Anthropic Messages endpoint and an OpenAI Responses endpoint, plus a browser chat and monitoring UI on localhost.

Several model sizes and quantizations are offered, and Apple Silicon Macs, Intel Arc, and some older GPUs are listed as experimental, built from source.

Key features

  • 01Runs the 125B-parameter Qwen3.8-Flash-Next model on a single consumer NVIDIA or AMD GPU.
  • 02Exposes OpenAI-compatible, Anthropic Messages and OpenAI Responses endpoints on localhost.
  • 03Installs by double-clicking a script on Windows or running a setup script on Linux.
  • 04Offers several model sizes so you can match quantization to your RAM.
  • 05Includes a browser chat UI and a live monitor of GPU, CPU and RAM usage.
  • 06Supports optional image input and multi-GPU setups.

Use cases

  • Running a capable local model for coding assistants such as Claude Code or Codex CLI without sending code to the cloud.
  • Keeping private documents or chat history on your own machine while still using a large model.
  • Pointing existing OpenAI-compatible apps at a local endpoint for offline or private use.
  • Benchmarking how a large MoE model performs on your own consumer GPU.

Topics

#llm#local-inference#gpu#quantization#openai-compatible#moe#llama-cpp#self-hosted

FAQ

What is Strata?
Local inference engine that runs the large Qwen3.8-Flash-Next model on consumer NVIDIA or AMD GPUs under Windows or Linux, with OpenAI and Anthropic compatible APIs. Strata is a local inference engine that runs Qwen3.8-Flash-Next, a very large model normally served from datacenter hardware, on an ordinary gaming PC. It requires an NVIDIA or AMD graphics card with at least 12 GB of VRAM, 32 GB or more of RAM, and Windows 10/11 or Linux. An installer detects the hardware, recommends a model size that fits the available memory, downloads the model and starts a local server.
Is Strata free to use? What license does it have?
Strata is open source under the MIT license. Check the license terms before using it in commercial products.
Is Strata still maintained?
Actively maintained. The last commit to Strata was on Oct 17, 2026.
What can I use Strata for?
Running a capable local model for coding assistants such as Claude Code or Codex CLI without sending code to the cloud. Keeping private documents or chat history on your own machine while still using a large model. Pointing existing OpenAI-compatible apps at a local endpoint for offline or private use. Benchmarking how a large MoE model performs on your own consumer GPU.
Where can I get Strata?
The source code and installation instructions are on GitHub: https://github.com/Niko1221/Strata

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects