Repolode
Submit
eagle-hq

eagle-hq/EAGLE-4

Inference code for EAGLE-4, a speculative decoding harness that speeds up LLM serving on SGLang, vLLM and TokenSpeed.

View on GitHubWebsite
Stars
229
Forks
6
Open issues
0
Last commit
8 hours ago

What is it?

EAGLE-4 is a speculative decoding system for accelerating large language model inference. A small draft model proposes candidate tokens, and the target model verifies them, with the goal of keeping the target model's output distribution. The draft adapts its speculative tree to batch size, hardware and workload.

This repository, eagle-harness, is a pip-installable package that plugs EAGLE-4 drafts into SGLang, vLLM and the experimental TokenSpeed engine through engine hooks, without patching the engine at runtime. Patched engines are distributed as pinned wheels and container images. The measured target pair is Qwen3.8-27B with its EAGLE-4 draft, on NVIDIA B300 and AMD MI355X hardware.

The project includes install and serve recipes, a calibration command for vLLM cost tables, and a support matrix reporting throughput against no-speculation baselines. Text-only requests are supported, and the repository is source-available under its own license rather than an OSI-approved one.

Key features

  • 01Speculative decoding with EAGLE-4 drafts for Qwen3.8-27B
  • 02Engine plugins for SGLang, vLLM and experimental TokenSpeed
  • 03Adaptive speculative tree sized from measured draft and verification costs
  • 04Pinned wheels and container images for NVIDIA B300 and AMD MI355X
  • 05Built-in environment check command, eagle-harness-check

Use cases

  • Lowering per-token latency when serving Qwen3.8-27B on multi-GPU NVIDIA B300 nodes
  • Improving throughput of a single-request or low-concurrency vLLM deployment
  • Running EAGLE-4 drafts on AMD MI355X with SGLang
  • Benchmarking speculative decoding against a no-speculation control

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/eagle-hq/EAGLE-4
  2. 2

    Load the version pins from the release settings file

    . EAGLE-4/docs/release.env
  3. 3

    Install the patched SGLang engine (CUDA)

    python -m pip install --extra-index-url "$EAGLE_CUDA_INDEX" --extra-index-url "$EAGLE_TORCH_CUDA_INDEX" "sglang==$EAGLE_SGLANG_VERSION" 'flashinfer-python>=0.7.2'
  4. 4

    Install eagle-harness from PyPI and run the check

    python -m pip install "eagle-harness==$EAGLE_VERSION"

Things to know

  • The license is source-available, not OSI-approved open source.
  • Commercial use of a competing offering needs a separate written license, and other commercial use is capped by a revenue threshold.
  • The published recipes target NVIDIA B300 or AMD MI355X GPUs, and the main example uses four GPUs.
  • TokenSpeed support is experimental and limited to CUDA.

Topics

#llm-inference#speculative-decoding#gpu#sglang#vllm#eagle#tokenspeed

FAQ

What is EAGLE-4?
Inference code for EAGLE-4, a speculative decoding harness that speeds up LLM serving on SGLang, vLLM and TokenSpeed. EAGLE-4 is a speculative decoding system for accelerating large language model inference. A small draft model proposes candidate tokens, and the target model verifies them, with the goal of keeping the target model's output distribution. The draft adapts its speculative tree to batch size, hardware and workload.
Is EAGLE-4 free to use? What license does it have?
EAGLE-4 is open source under the Other license. Check the license terms before using it in commercial products.
Is EAGLE-4 still maintained?
Actively maintained. The last commit to EAGLE-4 was on Oct 20, 2026.
What can I use EAGLE-4 for?
Lowering per-token latency when serving Qwen3.8-27B on multi-GPU NVIDIA B300 nodes Improving throughput of a single-request or low-concurrency vLLM deployment Running EAGLE-4 drafts on AMD MI355X with SGLang Benchmarking speculative decoding against a no-speculation control
How do I install EAGLE-4?
Clone the repository: git clone https://github.com/eagle-hq/EAGLE-4 Load the version pins from the release settings file: . EAGLE-4/docs/release.env Install the patched SGLang engine (CUDA): python -m pip install --extra-index-url "$EAGLE_CUDA_INDEX" --extra-index-url "$EAGLE_TORCH_CUDA_INDEX" "sglang==$EAGLE_SGLANG_VERSION" 'flashinfer-python>=0.7.2' Install eagle-harness from PyPI and run the check: python -m pip install "eagle-harness==$EAGLE_VERSION"
Where can I get EAGLE-4?
The source code and installation instructions are on GitHub: https://github.com/eagle-hq/EAGLE-4

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects