Repolode
Submit
ai-dynamo

ai-dynamo/dynamo

Datacenter-scale distributed inference serving framework that orchestrates vLLM, SGLang and TensorRT-LLM across multiple GPUs and nodes.

View on GitHubWebsite
Stars
8,382
Forks
1,755
Open issues
2,142
Last commit
4 hours ago

What is it?

Dynamo is an orchestration layer that sits above inference engines such as vLLM, SGLang and TensorRT-LLM rather than replacing them. It turns a cluster of GPUs into a coordinated inference system, handling disaggregated prefill and decode, KV-cache-aware request routing, multi-tier KV cache offloading, and SLA-driven autoscaling.

The project is written in Rust for the core runtime with Python for extensibility. It exposes an OpenAI-compatible HTTP frontend, and can be run from prebuilt containers, installed from PyPI, or deployed on Kubernetes with a custom resource for zero-config model deployment.

It targets LLM, reasoning, multimodal and video generation workloads where a single engine on a single GPU is no longer enough.

Key features

  • 01Disaggregated prefill and decode on independently scalable GPU pools
  • 02KV-aware routing that reduces redundant prefill computation
  • 03KV cache offloading across GPU, CPU, SSD and remote storage
  • 04SLA-based planner that autoscales replicas to latency targets
  • 05OpenAI-compatible API frontend
  • 06Supports SGLang, TensorRT-LLM and vLLM backends

Use cases

  • Serving large models across multiple GPUs or nodes in a Kubernetes cluster
  • Reducing time to first token for workloads with shared prompt prefixes
  • Running separate prefill and decode pools to match hardware to each phase
  • Deploying multimodal or video generation models with disaggregated encoding

Quick start

  1. 1

    Pull a prebuilt vLLM runtime container

    docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:
  2. 2

    Choose the SGLang or vLLM option in the README quick start and follow its steps

Things to know

  • Built for multi-GPU and multi-node datacenter deployments; a single model on one GPU is usually better served by the inference engine alone.
  • Some features are beta or partial, such as DGDR zero-config deploy and KVBM on SGLang.
  • The license is listed as NOASSERTION in metadata, so check the repository's license terms before use.

In collections

Run LLMs locallyRun and train AI on GPUsMicroservice frameworks & service infrastructure

Topics

#llm-inference#distributed-systems#kubernetes#disaggregated-serving#kv-cache#gpu#vllm#rust#diffusion#omni

FAQ

What is dynamo?
Datacenter-scale distributed inference serving framework that orchestrates vLLM, SGLang and TensorRT-LLM across multiple GPUs and nodes. Dynamo is an orchestration layer that sits above inference engines such as vLLM, SGLang and TensorRT-LLM rather than replacing them. It turns a cluster of GPUs into a coordinated inference system, handling disaggregated prefill and decode, KV-cache-aware request routing, multi-tier KV cache offloading, and SLA-driven autoscaling.
Is dynamo free to use? What license does it have?
dynamo is open source under the Other license. Check the license terms before using it in commercial products.
Is dynamo still maintained?
Actively maintained. The last commit to dynamo was on Oct 24, 2026.
What can I use dynamo for?
Serving large models across multiple GPUs or nodes in a Kubernetes cluster Reducing time to first token for workloads with shared prompt prefixes Running separate prefill and decode pools to match hardware to each phase Deploying multimodal or video generation models with disaggregated encoding
How do I install dynamo?
Pull a prebuilt vLLM runtime container: docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime: Choose the SGLang or vLLM option in the README quick start and follow its steps
Where can I get dynamo?
The source code and installation instructions are on GitHub: https://github.com/ai-dynamo/dynamo

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects