Repolode
Submit
intentee

intentee/paddler

Open-source LLM load balancer and serving platform that runs inference on your own CPU or GPU infrastructure using llama.cpp.

View on GitHubWebsite
Stars
1,690
Forks
97
Open issues
30
Last commit
18 hours ago

What is it?

Paddler is a self-hosted LLM load balancer and serving platform. It runs inference, embeddings, and multimodal models on your own infrastructure through a built-in llama.cpp engine, so applications can depend less on closed-source model APIs and per-token pricing.

It ships as a single binary with two deployable components: a balancer that distributes incoming requests, and agents that run inference slots. Agents can be added dynamically, which allows integration with autoscaling tools, and incoming requests can be buffered so a deployment can scale from zero hosts. Models can be swapped dynamically.

A built-in web admin panel lets you monitor the fleet, edit models and chat templates, and test prompts in a GUI. A beta desktop application targets casual use, such as pooling laptops and PCs into a local AI cluster.

Key features

  • 01Built-in llama.cpp inference engine for tokens and embeddings
  • 02LLM-aware load balancing across agents and slots
  • 03Dynamic agent registration for autoscaling integration
  • 04Request buffering that supports scaling from zero hosts
  • 05Dynamic model swapping
  • 06Web admin panel for monitoring, configuration, and testing
  • 07Observability metrics

Use cases

  • Running LLM inference and embeddings inside a product while keeping data on your own servers
  • Serving models to teams with strict privacy or compliance requirements
  • Pooling spare GPUs or PCs across an office into a local AI cluster
  • Replacing unpredictable per-token API costs with self-managed capacity

Quick start

  1. 1

    Obtain the paddler binary from the GitHub releases page

  2. 2

    Start the balancer

    paddler balancer --inference-addr 127.0.0.1:8061 --management-addr 127.0.0.1:8060 --web-admin-panel-addr 127.0.0.1:8062
  3. 3

    Start an agent with 4 slots

    paddler agent --management-addr 127.0.0.1:8060 --slots 4

Things to know

  • Building from source requires Rust MSRV 1.88.0 or newer.
  • The desktop application is still in beta.
  • Running models requires your own compute infrastructure, such as CPU or GPU hosts.

In collections

Self-hosted appsRun LLMs locallyRun and train AI on GPUs

Topics

#llm#llama-cpp#load-balancer#inference#self-hosted#gpu#embeddings#rust#ai#llamacpp

FAQ

What is paddler?
Open-source LLM load balancer and serving platform that runs inference on your own CPU or GPU infrastructure using llama.cpp. Paddler is a self-hosted LLM load balancer and serving platform. It runs inference, embeddings, and multimodal models on your own infrastructure through a built-in llama.cpp engine, so applications can depend less on closed-source model APIs and per-token pricing.
Is paddler free to use? What license does it have?
paddler is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is paddler still maintained?
Actively maintained. The last commit to paddler was on Nov 7, 2026.
What can I use paddler for?
Running LLM inference and embeddings inside a product while keeping data on your own servers Serving models to teams with strict privacy or compliance requirements Pooling spare GPUs or PCs across an office into a local AI cluster Replacing unpredictable per-token API costs with self-managed capacity
How do I install paddler?
Obtain the paddler binary from the GitHub releases page Start the balancer: paddler balancer --inference-addr 127.0.0.1:8061 --management-addr 127.0.0.1:8060 --web-admin-panel-addr 127.0.0.1:8062 Start an agent with 4 slots: paddler agent --management-addr 127.0.0.1:8060 --slots 4
Where can I get paddler?
The source code and installation instructions are on GitHub: https://github.com/intentee/paddler

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects