Repolode
Submit
algorithmicsuperintelligence

algorithmicsuperintelligence/optillm

An OpenAI API-compatible proxy that applies inference-time techniques like MoA, MCTS and best-of-N to improve LLM reasoning without training.

View on GitHub
Stars
4,338
Forks
399
Open issues
24
Last commit
5 days ago

What is it?

OptiLLM is an inference proxy that sits between your application and any OpenAI-compatible LLM endpoint. Clients send ordinary chat completion requests to OptiLLM, which runs extra compute at inference time, such as sampling multiple candidates, searching over plans, or verifying answers, before returning a response. No model training or fine-tuning is required.

Techniques are selected by prefixing the model name with a slug (for example `moa-gpt-4o-mini`), passing an `optillm_approach` field, or embedding a tag in the prompt. Approaches can be chained with `&` (pipeline) or `|` (parallel). The project also includes plugins for MCP tool access, PII privacy filtering, URL reading, code execution, structured JSON output, memory, and load balancing across providers.

OptiLLM works with OpenAI, Azure OpenAI, Cerebras, and other providers reachable through LiteLLM, as well as local llama.cpp or Ollama servers. It can also load HuggingFace models and LoRA adapters through its built-in inference server.

Key features

  • 01Drop-in OpenAI-compatible chat completions endpoint.
  • 02Selects techniques such as MoA, MCTS, best-of-N, and CePO via model prefixes or request fields.
  • 03Supports chaining or parallel running of approaches with & and | operators.
  • 04Routes to OpenAI, Azure, Cerebras, and 100+ providers through LiteLLM.
  • 05Offers plugins for MCP tools, PII privacy, URL reading, and code execution.
  • 06Includes a local inference server for HuggingFace models and LoRA adapters.

Use cases

  • Improving math or coding accuracy of a cheaper model without fine-tuning.
  • Adding reasoning-time compute to an existing app by changing only the base URL.
  • Connecting an LLM application to MCP servers for tools and files.
  • Running local llama.cpp or Ollama models with multi-sample reasoning methods.

Quick start

  1. 1

    Install OptiLLM from PyPI

    pip install optillm
  2. 2

    Set your OpenAI API key and start the server

    export OPENAI_API_KEY="your-key-here" && optillm
  3. 3

    Point any OpenAI client at http://localhost:8000/v1 and prefix the model name with moa- for Mixture of Agents

Things to know

  • Improvements depend on extra inference compute, so requests may be slower and cost more.
  • Some techniques need local models or specific providers, and the README's benchmark gains are reported for particular models and tasks.

In collections

Open-source AI agent frameworks & apps

Topics

#llm#inference#openai-api#prompt-engineering#agents#proxy-server#chain-of-thought#agent#agentic-ai#agentic-framework

FAQ

What is optillm?
An OpenAI API-compatible proxy that applies inference-time techniques like MoA, MCTS and best-of-N to improve LLM reasoning without training. OptiLLM is an inference proxy that sits between your application and any OpenAI-compatible LLM endpoint. Clients send ordinary chat completion requests to OptiLLM, which runs extra compute at inference time, such as sampling multiple candidates, searching over plans, or verifying answers, before returning a response. No model training or fine-tuning is required.
Is optillm free to use? What license does it have?
optillm is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is optillm still maintained?
Actively maintained. The last commit to optillm was on Nov 2, 2026.
What can I use optillm for?
Improving math or coding accuracy of a cheaper model without fine-tuning. Adding reasoning-time compute to an existing app by changing only the base URL. Connecting an LLM application to MCP servers for tools and files. Running local llama.cpp or Ollama models with multi-sample reasoning methods.
How do I install optillm?
Install OptiLLM from PyPI: pip install optillm Set your OpenAI API key and start the server: export OPENAI_API_KEY="your-key-here" && optillm Point any OpenAI client at http://localhost:8000/v1 and prefix the model name with moa- for Mixture of Agents
Where can I get optillm?
The source code and installation instructions are on GitHub: https://github.com/algorithmicsuperintelligence/optillm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects