Repolode
Submit
mnfst

mnfst/llm-gateway

Open-source LLM gateway that routes requests from AI agents and apps to many model providers through one OpenAI-compatible endpoint.

Open-source alternative to:OpenRouter

View on GitHubWebsite
Stars
7,595
Forks
522
Open issues
54
Last commit
yesterday

What is it?

Manifest LLM Gateway sits between your AI agents or apps and the model providers they call. You connect API keys, subscriptions, local models, or custom OpenAI/Anthropic-compatible endpoints, and every request goes through one OpenAI-compatible endpoint. Sending "model": "auto" lets Manifest choose the model for each query, which avoids locking an application to a single provider.

The gateway logs full request and error bodies, tracks spending per request, and supports cost notifications and limits. When a query fails, it can fall back to another model. Providers are listed as 35 built-in connections covering hosted APIs such as OpenAI, Anthropic, Google, and DeepSeek, along with local runtimes like Ollama, LM Studio, and llama.cpp.

It ships as a Docker image and can be deployed to Railway, Render, AWS, GCP, DigitalOcean, Fly.io and other platforms. Request recordings are stored in PostgreSQL and S3-compatible or persistent disk storage depending on the platform.

Key features

  • 01Single OpenAI-compatible endpoint for API keys, subscriptions, local models, and custom providers
  • 02Automatic model selection with "model": "auto"
  • 03Full request and error body logging
  • 04Per-request cost tracking with spending notifications and limits
  • 05Fallback to other models when a request fails

Use cases

  • Routing an AI agent across several providers without changing application code
  • Tracking and capping LLM spending across a team or project
  • Running a self-hosted gateway that mixes cloud APIs with local Ollama or llama.cpp models
  • Debugging agent requests by inspecting logged request and error bodies

Topics

#llm#gateway#openai-compatible#llm-router#cost-tracking#observability#self-hosted#docker#ai#ai-sdk

FAQ

What is llm-gateway?
Open-source LLM gateway that routes requests from AI agents and apps to many model providers through one OpenAI-compatible endpoint. Manifest LLM Gateway sits between your AI agents or apps and the model providers they call. You connect API keys, subscriptions, local models, or custom OpenAI/Anthropic-compatible endpoints, and every request goes through one OpenAI-compatible endpoint. Sending "model": "auto" lets Manifest choose the model for each query, which avoids locking an application to a single provider.
Is llm-gateway free to use? What license does it have?
llm-gateway is open source under the MIT license. Check the license terms before using it in commercial products.
Is llm-gateway still maintained?
Actively maintained. The last commit to llm-gateway was on Oct 21, 2026.
What can I use llm-gateway for?
Routing an AI agent across several providers without changing application code Tracking and capping LLM spending across a team or project Running a self-hosted gateway that mixes cloud APIs with local Ollama or llama.cpp models Debugging agent requests by inspecting logged request and error bodies
Where can I get llm-gateway?
The source code and installation instructions are on GitHub: https://github.com/mnfst/llm-gateway

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects