Repolode
Submit
llm-d

llm-d/llm-d-router

An intelligent entry point for LLM inference traffic that routes requests on Kubernetes using load, prefix-cache, and priority signals.

View on GitHubWebsite
Stars
455
Forks
517
Open issues
374
Last commit
15 hours ago

What is it?

llm-d Router is a request routing layer for LLM inference. Its Endpoint Picker (EPP) evaluates each incoming request against the current state of a backend pool, considering KV-cache locality, current load, and priority to choose the best model server. It plugs into Envoy-based proxies through the ext-proc protocol, so placement decisions happen in the data plane.

The router can run in standalone mode with a self-managed Envoy proxure deployed as a sidecar or as a separate scalable service, configured through a Helm chart. For production, it can also run in Gateway mode, where an InferencePool is referenced by an HTTPRoute on a shared Gateway API gateway, including integrations with Istio, AgentGateway, and Google Cloud's Application Load Balancer.

The repository also includes InferenceObjective and InferenceModelRewrite APIs for scheduling goals and model name rewriting, which support A/B tests and canary rollouts. A disaggregation sidecar coordinates Prefill/Decode and Encode/Prefill/Decode workflows with specialized workers.

Key features

  • 01Routes requests using KV-cache locality, current load, and priority signals.
  • 02Integrates with Envoy through the ext-proc protocol.
  • 03Supports standalone deployment via Helm, with sidecar or service proxy topologies.
  • 04Works with Kubernetes Gateway API, including Istio and cloud load balancers.
  • 05Provides model name rewriting for A/B testing and canary rollouts.
  • 06Coordinates P/D and E/P/D disaggregated inference workflows.

Use cases

  • Routing LLM inference traffic across multiple model server replicas on Kubernetes.
  • Improving prefix-cache hit rates by sending related requests to the same backend.
  • Running canary rollouts or A/B tests by rewriting model names at the gateway.
  • Serving disaggregated Prefill/Decode inference pipelines.

Quick start

  1. 1

    Read the Helm chart documentation for standalone deployment on Kubernetes

Things to know

  • Requires a Kubernetes cluster; it is designed to route inference traffic on Kubernetes.
  • If you install and configure Envoy yourself, the only supported request_body_mode and response_body_mode is FULL_DUPLEX_STREAMED.

In collections

Run LLMs locally

Topics

#llm-inference#kubernetes#gateway-api#envoy#load-balancing#llm-routing#ext-proc#ai#inference#networking

FAQ

What is llm-d-router?
An intelligent entry point for LLM inference traffic that routes requests on Kubernetes using load, prefix-cache, and priority signals. llm-d Router is a request routing layer for LLM inference. Its Endpoint Picker (EPP) evaluates each incoming request against the current state of a backend pool, considering KV-cache locality, current load, and priority to choose the best model server. It plugs into Envoy-based proxies through the ext-proc protocol, so placement decisions happen in the data plane.
Is llm-d-router free to use? What license does it have?
llm-d-router is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is llm-d-router still maintained?
Actively maintained. The last commit to llm-d-router was on Nov 5, 2026.
What can I use llm-d-router for?
Routing LLM inference traffic across multiple model server replicas on Kubernetes. Improving prefix-cache hit rates by sending related requests to the same backend. Running canary rollouts or A/B tests by rewriting model names at the gateway. Serving disaggregated Prefill/Decode inference pipelines.
How do I install llm-d-router?
Read the Helm chart documentation for standalone deployment on Kubernetes
Where can I get llm-d-router?
The source code and installation instructions are on GitHub: https://github.com/llm-d/llm-d-router

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects