Repolode
Submit
Edge0-AI

Edge0-AI/Edge0

An open-source framework that runs large sparse-MoE models on consumer devices by streaming expert weights from SSD with trained routing prediction.

View on GitHub
Stars
9,113
Forks
1,405
Open issues
8
Last commit
23 hours ago

What is it?

edge0 is a streaming inference framework for sparse Mixture-of-Experts (MoE) language models. Instead of holding all expert weights in memory, it streams expert weights from storage on demand, so peak memory is bounded by the set of active experts rather than the total parameter count. The project provides two released model tiers, a 35B-A3B model and an 8B-A1B model, both 4-bit quantized.

To hide the cost of loading experts from disk, a trained prerouter head predicts the next step's expert routing, so loads overlap with the forward pass. Quantization loss is reduced with Recover-LoRA adapters trained by distillation from the full-precision teacher; the adapters stay unmerged so one read-only base can serve several adapter sets.

The Python framework currently runs on macOS with Apple Silicon via MLX and exposes an OpenAI-compatible chat completions endpoint. Native engines for macOS, iOS, Android and Windows are included in the repository, and a unified access layer is planned for Q4 2026.

Key features

  • 01Streams expert weights from SSD on demand, bounding memory by active experts.
  • 02Trained prerouter predicts expert routing one step ahead to overlap loads with compute.
  • 03Recover-LoRA adapters recover most int4 quantization loss and stay unmerged.
  • 04OpenAI-compatible /v1/chat/completions server and CLI on macOS with Apple Silicon.
  • 05Native engine directories for macOS, iOS, Android and Windows.

Use cases

  • Running a 35B-class MoE model on a Mac with limited unified memory.
  • Experimenting with on-device LLM inference on iPhone, Android or Windows PCs.
  • Serving a local OpenAI-compatible chat endpoint for development tools on Apple Silicon.
  • Studying expert offloading and routing prediction for memory-constrained inference.

Quick start

  1. 1

    Create a virtual environment and install the Python package (macOS on Apple Silicon)

    python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'
  2. 2

    Download the 8B model (about 4.2 GB) into models/edge0-8b

    .venv/bin/huggingface-cli download Edge0/Edge0-8B-A1B-preview --local-dir models/edge0-8b
  3. 3

    Run the quick demo on the downloaded model

    edge0 demo models/edge0-8b

Things to know

  • The Python framework only runs on macOS with Apple Silicon; the CUDA backend is not yet available.
  • The released models are labeled preview.
  • The 35B checkpoint is about 23 GB on disk.

Featured in Repolode Weekly

In collections

Run LLMs locallyOpen-source Mac apps

Topics

#llm-inference#mixture-of-experts#model-quantization#on-device-ai#apple-silicon#mlx#lora#edge-computing

FAQ

What is Edge0?
An open-source framework that runs large sparse-MoE models on consumer devices by streaming expert weights from SSD with trained routing prediction. edge0 is a streaming inference framework for sparse Mixture-of-Experts (MoE) language models. Instead of holding all expert weights in memory, it streams expert weights from storage on demand, so peak memory is bounded by the set of active experts rather than the total parameter count. The project provides two released model tiers, a 35B-A3B model and an 8B-A1B model, both 4-bit quantized.
Is Edge0 free to use? What license does it have?
Edge0 is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is Edge0 still maintained?
Actively maintained. The last commit to Edge0 was on Nov 6, 2026.
What can I use Edge0 for?
Running a 35B-class MoE model on a Mac with limited unified memory. Experimenting with on-device LLM inference on iPhone, Android or Windows PCs. Serving a local OpenAI-compatible chat endpoint for development tools on Apple Silicon. Studying expert offloading and routing prediction for memory-constrained inference.
How do I install Edge0?
Create a virtual environment and install the Python package (macOS on Apple Silicon): python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]' Download the 8B model (about 4.2 GB) into models/edge0-8b: .venv/bin/huggingface-cli download Edge0/Edge0-8B-A1B-preview --local-dir models/edge0-8b Run the quick demo on the downloaded model: edge0 demo models/edge0-8b
Where can I get Edge0?
The source code and installation instructions are on GitHub: https://github.com/Edge0-AI/Edge0

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects