Repolode
Submit
vllm-project

vllm-project/semantic-router

An open, programmable decision layer that selects or combines LLMs for each request across local, private, and cloud backends, with one API endpoint.

View on GitHubWebsite
Stars
6,412
Forks
1,111
Open issues
878
Last commit
16 hours ago

What is it?

vLLM Semantic Router sits between an agent or application and a set of model backends. Instead of sending every call to one model, it exposes a single inference endpoint (for example the public model ID vllm-sr/auto) and decides, per call, which configured model or combination of models should answer, based on policies the operator defines.

The routing policy covers quality, latency, and cost priorities, as well as where calls are allowed to run, such as edge, private, or cloud compute. The agent harness keeps its own loop, tools, and task state; the router only handles model and backend selection. Its installer, playground, and Hugging Face model collections are part of the project's published tooling.

The project is written mainly in Go and is aimed at teams running several LLMs who want routing decisions to be configurable rather than hard-coded in application code. It is licensed under Apache-2.0 and includes Kubernetes-related topics in its repository metadata.

Key features

  • 01Exposes one inference endpoint that routes requests across multiple configured models.
  • 02Selects or combines models per call based on operator-defined policy.
  • 03Lets quality, latency, and cost priorities drive routing decisions.
  • 04Keeps calls within approved locations across edge, private, and cloud compute.
  • 05Provides a Hugging Face model collection of routing models (Decision and Vela series).

Use cases

  • Routing an agent harness to different models depending on task difficulty or cost.
  • Keeping sensitive requests on private or on-premises compute while sending others to cloud models.
  • Running a mixture-of-models setup across several vLLM-served backends behind one endpoint.

Quick start

  1. 1

    Run the official install script from the README

  2. 2

    Point your agent harness at the Router endpoint using model ID vllm-sr/auto

  3. 3

    Optionally try the online playground

Topics

#llm#ai-gateway#inference#llm-routing#kubernetes#guardrails#vllm#mixture-of-models#llmrouter#pytorch

FAQ

What is semantic-router?
An open, programmable decision layer that selects or combines LLMs for each request across local, private, and cloud backends, with one API endpoint. vLLM Semantic Router sits between an agent or application and a set of model backends. Instead of sending every call to one model, it exposes a single inference endpoint (for example the public model ID vllm-sr/auto) and decides, per call, which configured model or combination of models should answer, based on policies the operator defines.
Is semantic-router free to use? What license does it have?
semantic-router is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is semantic-router still maintained?
Actively maintained. The last commit to semantic-router was on Nov 6, 2026.
What can I use semantic-router for?
Routing an agent harness to different models depending on task difficulty or cost. Keeping sensitive requests on private or on-premises compute while sending others to cloud models. Running a mixture-of-models setup across several vLLM-served backends behind one endpoint.
How do I install semantic-router?
Run the official install script from the README Point your agent harness at the Router endpoint using model ID vllm-sr/auto Optionally try the online playground
Where can I get semantic-router?
The source code and installation instructions are on GitHub: https://github.com/vllm-project/semantic-router

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects