Repolode
Submit
mylxsw

mylxsw/llm-gateway

Self-hosted LLM gateway that proxies OpenAI, Anthropic and compatible APIs with routing, failover, request logging and a web dashboard.

Open-source alternative to:OpenRouterPortkey

View on GitHubWebsite
Stars
76
Forks
20
Open issues
1
Last commit
18 days ago

What is it?

Squirrel is a proxy service that sits between your applications and LLM providers. Applications point their OpenAI or Anthropic SDK at the gateway, and it forwards requests to configured upstream providers, so provider credentials, model names and routing rules are managed in one place.

Routing supports round-robin, priority, weight-based and cost-based strategies, along with rules based on model name, headers, message content or token count. When an upstream returns server errors, requests are retried and can fail over to backup providers. It also converts between OpenAI Chat, OpenAI Responses and Anthropic Messages formats, including streaming.

Every request is logged with token counts, latency and full request and response bodies, with separate retention for heavy payloads. A Next.js dashboard manages providers, model mappings and API keys, and an MCP endpoint lets AI agents inspect or manage the gateway. It runs on SQLite by default or PostgreSQL, and can be deployed with Docker.

Key features

  • 01Exposes OpenAI Chat, Responses, Embeddings, audio and image endpoints plus Anthropic Messages.
  • 02Converts requests and streaming responses between OpenAI Chat, OpenAI Responses and Anthropic formats.
  • 03Routes by round-robin, priority, weight or lowest cost, with rule-based matching.
  • 04Retries server errors and fails over to backup providers.
  • 05Logs token usage, latency and full request/response bodies with redaction of sensitive data.
  • 06Provides a web dashboard and an MCP interface for management.

Use cases

  • Giving several internal applications one endpoint and one API key for multiple LLM providers.
  • Adding failover so an outage at one provider does not break an AI feature.
  • Tracking token usage and cost per API key, model or provider.
  • Pointing existing OpenAI or Anthropic SDK code at self-hosted or local models such as Ollama or vLLM.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/mylxsw/llm-gateway.git
  2. 2

    Enter the project directory and set ADMIN_USERNAME and ADMIN_PASSWORD before starting

    cd llm-gateway
  3. 3

    Start the services with Docker Compose (dashboard at http://localhost:8000)

    docker compose -f docker-compose.prod.yml up -d

Things to know

  • Admin APIs are open by default if ADMIN_USERNAME and ADMIN_PASSWORD are not set, so credentials must be configured.
  • The project is self-hosted and requires setting up providers, API keys and an encryption key for production use.
  • The metadata lists no license, although the README badge says MIT.

Topics

#llm#llm-gateway#proxy#openai#anthropic#load-balancing#failover#api-gateway

FAQ

What is llm-gateway?
Self-hosted LLM gateway that proxies OpenAI, Anthropic and compatible APIs with routing, failover, request logging and a web dashboard. Squirrel is a proxy service that sits between your applications and LLM providers. Applications point their OpenAI or Anthropic SDK at the gateway, and it forwards requests to configured upstream providers, so provider credentials, model names and routing rules are managed in one place.
Is llm-gateway free to use? What license does it have?
llm-gateway has no declared license. Its code is public, but without a license you have no explicit permission to reuse it; ask the author before relying on it.
Is llm-gateway still maintained?
Actively maintained. The last commit to llm-gateway was on Oct 4, 2026.
What can I use llm-gateway for?
Giving several internal applications one endpoint and one API key for multiple LLM providers. Adding failover so an outage at one provider does not break an AI feature. Tracking token usage and cost per API key, model or provider. Pointing existing OpenAI or Anthropic SDK code at self-hosted or local models such as Ollama or vLLM.
How do I install llm-gateway?
Clone the repository: git clone https://github.com/mylxsw/llm-gateway.git Enter the project directory and set ADMIN_USERNAME and ADMIN_PASSWORD before starting: cd llm-gateway Start the services with Docker Compose (dashboard at http://localhost:8000): docker compose -f docker-compose.prod.yml up -d
Where can I get llm-gateway?
The source code and installation instructions are on GitHub: https://github.com/mylxsw/llm-gateway

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects