Repolode
Submit
BerriAI

BerriAI/litellm

Open source AI Gateway and Python SDK that calls 100+ LLM providers through a single OpenAI-format interface.

View on GitHubWebsite
Stars
61,636
Forks
12,501
Open issues
5,530
Last commit
6 hours ago

What is it?

LiteLLM lets developers call many LLM providers, including OpenAI, Anthropic, Gemini, Bedrock, Azure and Vertex AI, through one OpenAI-compatible interface. It can be used as a Python library in application code, or deployed as a self-hosted proxy server that serves as a central gateway for a team or organization.

The proxy adds operational features such as virtual keys, spend tracking per project or user, guardrails, load balancing and fallback routing, logging callbacks, and an admin dashboard. It also exposes endpoints for chat completions, embeddings, images, audio, batches and rerank, and can route calls to A2A agents and MCP servers.

The project provides a Rust core with a Python SDK, and Terraform modules for deploying the proxy on AWS and GCP.

Key features

  • 01Calls 100+ LLM providers using the OpenAI request format.
  • 02Self-hosted proxy with virtual keys and per-project spend tracking.
  • 03Load balancing, retries and fallback routing across multiple deployments.
  • 04Guardrails and logging callbacks for observability.
  • 05Connects A2A agents and MCP servers through the gateway.

Use cases

  • Giving a team one internal endpoint for multiple LLM providers with per-user keys and budgets.
  • Switching between OpenAI and Anthropic models in application code without rewriting API calls.
  • Deploying an LLM gateway on AWS or GCP using the provided Terraform modules.

Quick start

  1. 1

    Install the LiteLLM proxy server with uv

    uv tool install 'litellm[proxy]'
  2. 2

    Start the proxy with a model, then call it at http://0.0.0.0:4000

    litellm --model gpt-4o

Things to know

  • The license is not clearly identified (metadata shows NOASSERTION), so check the license terms before use.
  • Calling providers requires your own API keys for each provider.
  • Some advertised features, such as the enterprise tier, are separate from the open source project.

Topics

#llm#ai-gateway#openai-compatible#llm-proxy#load-balancing#python#anthropic#azure-openai#bedrock#gateway

FAQ

What is litellm?
Open source AI Gateway and Python SDK that calls 100+ LLM providers through a single OpenAI-format interface. LiteLLM lets developers call many LLM providers, including OpenAI, Anthropic, Gemini, Bedrock, Azure and Vertex AI, through one OpenAI-compatible interface. It can be used as a Python library in application code, or deployed as a self-hosted proxy server that serves as a central gateway for a team or organization.
Is litellm free to use? What license does it have?
litellm is open source under the Other license. Check the license terms before using it in commercial products.
Is litellm still maintained?
Actively maintained. The last commit to litellm was on Oct 20, 2026.
What can I use litellm for?
Giving a team one internal endpoint for multiple LLM providers with per-user keys and budgets. Switching between OpenAI and Anthropic models in application code without rewriting API calls. Deploying an LLM gateway on AWS or GCP using the provided Terraform modules.
How do I install litellm?
Install the LiteLLM proxy server with uv: uv tool install 'litellm[proxy]' Start the proxy with a model, then call it at http://0.0.0.0:4000: litellm --model gpt-4o
Where can I get litellm?
The source code and installation instructions are on GitHub: https://github.com/BerriAI/litellm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects