Repolode
Submit
vllm-project

vllm-project/speculators

A Python library for training speculative decoding draft models that deploy directly into the vLLM inference engine.

View on GitHubWebsite
Stars
920
Forks
271
Open issues
204
Last commit
yesterday

What is it?

Speculators is a library for training speculative decoding draft models that run directly in LLM inference engines such as vLLM. Speculative decoding speeds up generation by having a smaller draft model propose several tokens that the large base model then verifies in a single forward pass. Accepted tokens match what the base model would have produced, so output quality is unchanged.

The project provides an end-to-end pipeline: generating hidden states with vLLM and saving them to disk, training single- and multi-layer draft models, and storing the results in a Hugging Face-compatible format. It supports training for MoE, non-MoE, and vision-language models, and includes tools for converting draft models from external research repositories into its standard format.

Trained speculators can be served with a single `vllm serve` command. Supported algorithms mentioned in the project include EAGLE-3, DFlash, DSpark and P-EAGLE, and multi-node online training is available through pluggable hidden-state transfer backends.

Key features

  • 01Generates hidden states with vLLM and saves them to disk for draft model training.
  • 02Trains single- and multi-layer draft models, including for MoE and vision-language models.
  • 03Stores speculators in a Hugging Face-compatible, extensible format.
  • 04Converts draft models from external research repositories into the standard format.
  • 05Deploys trained speculators directly to vLLM with `vllm serve`.
  • 06Supports EAGLE-3, DFlash, DSpark and P-EAGLE training algorithms.

Use cases

  • Speeding up serving of a large LLM in vLLM by training a matching draft model.
  • Training a draft model on domain-specific data to better match a production workload.
  • Finetuning the native MTP heads of models such as Qwen3-Next.
  • Benchmarking speculative decoding configurations with GuideLLM.

Quick start

  1. 1

    Install from PyPI

    pip install speculators
  2. 2

    Verify the installation

    speculators --version
  3. 3

    Serve a trained speculator model with vLLM

    vllm serve RedHatAI/Qwen3-8B-speculator.eagle3

Things to know

  • Requires Linux or macOS with Python 3.11 or higher.
  • Running served speculator models requires vLLM, which the README does not install for you.

In collections

Run LLMs locally

Topics

#speculative-decoding#llm-inference#vllm#draft-model#model-training#llm

FAQ

What is speculators?
A Python library for training speculative decoding draft models that deploy directly into the vLLM inference engine. Speculators is a library for training speculative decoding draft models that run directly in LLM inference engines such as vLLM. Speculative decoding speeds up generation by having a smaller draft model propose several tokens that the large base model then verifies in a single forward pass. Accepted tokens match what the base model would have produced, so output quality is unchanged.
Is speculators free to use? What license does it have?
speculators is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is speculators still maintained?
Actively maintained. The last commit to speculators was on Oct 23, 2026.
What can I use speculators for?
Speeding up serving of a large LLM in vLLM by training a matching draft model. Training a draft model on domain-specific data to better match a production workload. Finetuning the native MTP heads of models such as Qwen3-Next. Benchmarking speculative decoding configurations with GuideLLM.
How do I install speculators?
Install from PyPI: pip install speculators Verify the installation: speculators --version Serve a trained speculator model with vLLM: vllm serve RedHatAI/Qwen3-8B-speculator.eagle3
Where can I get speculators?
The source code and installation instructions are on GitHub: https://github.com/vllm-project/speculators

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects