Repolode
Submit
bentoml

bentoml/OpenLLM

Runs open-source LLMs such as Llama and Qwen as OpenAI-compatible API endpoints with a single command, with a built-in chat UI.

Open-source alternative to:OpenAI API

View on GitHubWebsite
Stars
12,586
Forks
847
Open issues
21
Last commit
5 days ago

What is it?

OpenLLM is a Python tool for self-hosting open-source large language models. You start a model with a command such as `openllm serve llama3.2:1b`, and it exposes an OpenAI-compatible API on localhost port 3000, so existing OpenAI client code and tools like LlamaIndex can point at it with only a base URL change.

The project ships a catalog of supported models, including Llama 3.x, Mistral, Qwen 2.5, Gemma and DeepSeek, and the server includes a chat UI at the /chat path. Gated models require a Hugging Face token, and OpenLLM does not store model weights itself. A CLI command, `openllm run`, opens a chat session in the terminal.

Models are organized in model repositories. Users can add a custom public repository, with models packaged as Bentos using BentoML, and the project describes building Docker and Kubernetes deployments from them.

Key features

  • 01Serves open-source LLMs as OpenAI-compatible API endpoints.
  • 02Starts a model server with a single `openllm serve` command.
  • 03Includes a built-in chat UI at /chat.
  • 04Provides a terminal chat command via `openllm run`.
  • 05Supports model repositories, including custom public ones.
  • 06Builds models into Bentos for Docker and Kubernetes deployment.

Use cases

  • Hosting a Llama or Qwen model locally behind an OpenAI-style API for an existing app.
  • Testing open-source models with OpenAI SDK code without changing the client.
  • Deploying a custom or fine-tuned model to Docker or Kubernetes using BentoML packaging.

Quick start

  1. 1

    Install OpenLLM with pip

    pip install openllm
  2. 2

    Explore OpenLLM interactively

    openllm hello
  3. 3

    Set your Hugging Face token for gated models

    export HF_TOKEN=<your token>
  4. 4

    Start an LLM server with a model

    openllm serve llama3.2:1b

Things to know

  • Gated models such as Llama require a Hugging Face token and access approval.
  • Larger models need substantial GPU memory, for example 80G GPUs for some entries.

In collections

Self-hosted apps

Topics

#llm#llm-serving#openai-compatible#self-hosted#inference#model-serving#python#bentoml#fine-tuning#llama

FAQ

What is OpenLLM?
Runs open-source LLMs such as Llama and Qwen as OpenAI-compatible API endpoints with a single command, with a built-in chat UI. OpenLLM is a Python tool for self-hosting open-source large language models. You start a model with a command such as `openllm serve llama3.2:1b`, and it exposes an OpenAI-compatible API on localhost port 3000, so existing OpenAI client code and tools like LlamaIndex can point at it with only a base URL change.
Is OpenLLM free to use? What license does it have?
OpenLLM is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is OpenLLM still maintained?
Actively maintained. The last commit to OpenLLM was on Nov 2, 2026.
What can I use OpenLLM for?
Hosting a Llama or Qwen model locally behind an OpenAI-style API for an existing app. Testing open-source models with OpenAI SDK code without changing the client. Deploying a custom or fine-tuned model to Docker or Kubernetes using BentoML packaging.
How do I install OpenLLM?
Install OpenLLM with pip: pip install openllm Explore OpenLLM interactively: openllm hello Set your Hugging Face token for gated models: export HF_TOKEN=<your token> Start an LLM server with a model: openllm serve llama3.2:1b
Where can I get OpenLLM?
The source code and installation instructions are on GitHub: https://github.com/bentoml/OpenLLM

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects