Repolode
Submit
FlashML-org

FlashML-org/FreeToken

An edge-native MoE serving engine that runs large open-weight models like DeepSeek-V4 and Qwen3.6 on consumer GPUs by splitting work between GPU and CPU memory.

Open-source alternative to:LM Studio

View on GitHubWebsite
Stars
14,575
Forks
1,468
Open issues
436
Last commit
18 hours ago

What is it?

FreeToken is a serving engine for running frontier-scale Mixture-of-Experts models on personal hardware such as gaming PCs and laptops. It coordinates GPUs, CPUs, host memory and interconnects as one inference pool, using bandwidth-adaptive CPU-GPU co-execution, streamed prefill, and a global LRU cache for experts.

The engine targets models such as DeepSeek-V4-Flash, Qwen3.6-35B-A3B and GLM-5.2, with support for MXFP4, NVFP4, FP8 and BF16 quantization. It exposes Anthropic- and OpenAI-compatible APIs so coding and tool-calling agents such as Codex, Claude Code and OpenCode can connect to local models.

FreeToken also offers semantic anchor checkpoints for recurrent state and KV caches, so agentic context edits avoid recomputing prior context, and it can reallocate VRAM between expert cache and KV memory at runtime. A Windows and Linux desktop app and a pip-installable CLI are provided, with native support for NVIDIA RTX 30, 40 and 50 series GPUs.

Key features

  • 01Runs large MoE models on consumer GPUs with CPU-GPU co-execution and expert caching.
  • 02Supports quantized formats including MXFP4, NVFP4, FP8 and BF16.
  • 03Provides Anthropic- and OpenAI-compatible APIs for coding and tool-calling agents.
  • 04Reallocates VRAM between expert cache and KV memory without restarting the engine.
  • 05Ships a desktop GUI for Windows and Linux plus a pip-installable CLI.

Use cases

  • Running frontier open-weight MoE models locally on a gaming PC or workstation GPU.
  • Connecting local models to coding agents such as Claude Code or Codex.
  • Experimenting with large models without cloud inference costs or data leaving the machine.

Topics

#moe#llm-inference#edge-ai#local-llm#quantization#gpu#openai-compatible#deepseek-v4#edge#glm

FAQ

What is FreeToken?
An edge-native MoE serving engine that runs large open-weight models like DeepSeek-V4 and Qwen3.6 on consumer GPUs by splitting work between GPU and CPU memory. FreeToken is a serving engine for running frontier-scale Mixture-of-Experts models on personal hardware such as gaming PCs and laptops. It coordinates GPUs, CPUs, host memory and interconnects as one inference pool, using bandwidth-adaptive CPU-GPU co-execution, streamed prefill, and a global LRU cache for experts.
Is FreeToken free to use? What license does it have?
FreeToken is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is FreeToken still maintained?
Actively maintained. The last commit to FreeToken was on Oct 18, 2026.
What can I use FreeToken for?
Running frontier open-weight MoE models locally on a gaming PC or workstation GPU. Connecting local models to coding agents such as Claude Code or Codex. Experimenting with large models without cloud inference costs or data leaving the machine.
Where can I get FreeToken?
The source code and installation instructions are on GitHub: https://github.com/FlashML-org/FreeToken

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects