Repolode
Submit
mostlygeek

mostlygeek/llama-swap

A lightweight Go proxy that hot-swaps local AI model servers such as llama.cpp and vLLM behind one OpenAI- and Anthropic-compatible endpoint.

View on GitHub
Stars
6,200
Forks
542
Open issues
116
Last commit
yesterday

What is it?

llama-swap runs multiple local generative AI models on one machine and switches between them on demand. It sits in front of inference servers such as llama.cpp's llama-server, vLLM, stable-diffusion.cpp and ComfyUI, and when a request names a model, it starts the matching server and stops the one that was running before.

The project ships as a single binary driven by one YAML file, where each model ID maps to a shell command that launches its server. It exposes OpenAI-style chat, completion, embedding, audio and image endpoints, Anthropic message endpoints, and llama-server-specific routes. Optional features include a web UI with a playground and log streaming, API keys, profiles, TTL-based automatic unloading, concurrent models through a matrix setting, and Docker or Podman containers.

Key features

  • 01Single binary and one configuration file with no external dependencies.
  • 02On-demand swapping between llama.cpp, vLLM, stable-diffusion.cpp, ComfyUI and other OpenAI-compatible servers.
  • 03Exposes OpenAI, Anthropic and llama-server compatible endpoints behind one address.
  • 04Built-in web UI with a playground, token metrics and manual load/unload controls.
  • 05Real-time log streaming and Prometheus metrics.
  • 06API key access control and automatic unloading after a configurable TTL.

Use cases

  • Running several local chat, coding and embedding models on one GPU and switching between them by model name.
  • Giving local tools an OpenAI-compatible endpoint that can serve different backends.
  • Hosting a shared local inference server with API keys and monitoring for a small team.

Quick start

  1. 1

    Install via Homebrew (macOS/Linux)

    brew tap mostlygeek/llama-swap
  2. 2

    Install the llama-swap formula

    brew install llama-swap
  3. 3

    Start llama-swap with your config file

    llama-swap --config path/to/config.yaml --listen localhost:8080

Things to know

  • Requires you to write a YAML config that points to your own model server commands and model files.

In collections

Run LLMs locally

Topics

#llm#llama-cpp#openai-api#model-serving#local-ai#proxy#golang#llama#llamacpp#localllama

FAQ

What is llama-swap?
A lightweight Go proxy that hot-swaps local AI model servers such as llama.cpp and vLLM behind one OpenAI- and Anthropic-compatible endpoint. llama-swap runs multiple local generative AI models on one machine and switches between them on demand. It sits in front of inference servers such as llama.cpp's llama-server, vLLM, stable-diffusion.cpp and ComfyUI, and when a request names a model, it starts the matching server and stops the one that was running before.
Is llama-swap free to use? What license does it have?
llama-swap is open source under the MIT license. Check the license terms before using it in commercial products.
Is llama-swap still maintained?
Actively maintained. The last commit to llama-swap was on Nov 6, 2026.
What can I use llama-swap for?
Running several local chat, coding and embedding models on one GPU and switching between them by model name. Giving local tools an OpenAI-compatible endpoint that can serve different backends. Hosting a shared local inference server with API keys and monitoring for a small team.
How do I install llama-swap?
Install via Homebrew (macOS/Linux): brew tap mostlygeek/llama-swap Install the llama-swap formula: brew install llama-swap Start llama-swap with your config file: llama-swap --config path/to/config.yaml --listen localhost:8080
Where can I get llama-swap?
The source code and installation instructions are on GitHub: https://github.com/mostlygeek/llama-swap

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects