smg-project/smg

What is it?

Key features

• 01Unified OpenAI, Anthropic and Gemini-style API endpoints across self-hosted and cloud backends.
• 02Cache-aware routing plus nine other load-balancing policies.
• 03Native gRPC pipeline with prefill/decode and encode disaggregation.
• 04Kubernetes pod discovery and high-availability mesh for multi-node deployments.
• 05MCP tool execution, tool-call and reasoning parsers for many open models.
• 06Prometheus metrics, OpenTelemetry tracing, and WebAssembly plugins.

Use cases

• Load-balancing requests across a fleet of vLLM or SGLang GPU servers.
• Exposing one OpenAI-compatible endpoint that fronts both local engines and hosted cloud models.
• Adding authentication, multi-tenant controls and chat history to an LLM service.
• Running a routed inference platform on Kubernetes with automatic worker discovery.

Quick start

1. 1

   Install via Docker

   docker pull lightseekorg/smg:latest
2. 2

   Install via Python

   pip install smg
3. 3

   Launch the gateway against your inference workers

   smg launch --worker-urls http://localhost:8000
4. 4

   Send a chat request to the gateway

   curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "llama3", "messages": [{"role": "user", "content": "Hello!"}]}'

Things to know

• !Running it requires at least one existing inference worker or cloud provider to route to.
• !The README does not describe a stable release status, so expect possible changes between versions.

Topics

FAQ

What is smg?

Engine-agnostic LLM gateway in Rust that routes OpenAI- and Anthropic-compatible traffic across self-hosted inference engines and cloud model providers. SMG (Shepherd Model Gateway) is a model-routing gateway that sits in front of LLM inference servers and cloud providers, exposing them behind one unified endpoint. It manages worker lifecycle, balances traffic across vLLM, SGLang, TensorRT-LLM, TokenSpeed, MLX and any OpenAI-compatible server, and can also forward requests to OpenAI, Anthropic, Gemini, xAI, AWS Bedrock and Azure OpenAI.

Is smg free to use? What license does it have?

smg is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.

Is smg still maintained?

Actively maintained. The last commit to smg was on Oct 19, 2026.

What can I use smg for?

Load-balancing requests across a fleet of vLLM or SGLang GPU servers. Exposing one OpenAI-compatible endpoint that fronts both local engines and hosted cloud models. Adding authentication, multi-tenant controls and chat history to an LLM service. Running a routed inference platform on Kubernetes with automatic worker discovery.

How do I install smg?

Install via Docker: docker pull lightseekorg/smg:latest Install via Python: pip install smg Launch the gateway against your inference workers: smg launch --worker-urls http://localhost:8000 Send a chat request to the gateway: curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "llama3", "messages": [{"role": "user", "content": "Hello!"}]}'

Where can I get smg?

The source code and installation instructions are on GitHub: https://github.com/smg-project/smg