Repolode
Submit
flitai-io

flitai-io/gpuworks

A lightweight GPU management and Model-as-a-Service platform that pools servers and clouds with mixed accelerators behind one OpenAI-compatible endpoint.

View on GitHub
Stars
141
Forks
41
Open issues
2
Last commit
24 hours ago

What is it?

GPUWorks is a GPU management and Model-as-a-Service (MaaS) platform aimed at teams that run LLMs and other models on a mix of hardware. You join bare-metal servers, cloud instances, VMs or a Mac to one cluster, choose a model, and the platform estimates the GPU memory it needs, picks suitable GPUs, starts an inference engine such as vLLM or SGLang, and exposes it through a single OpenAI-compatible endpoint.

The control plane is one static Go binary with an embedded web UI and keeps its state in a SQLite file. Node agents dial out over WebSocket, so GPU nodes do not need inbound ports. NVIDIA, AMD, Huawei Ascend, T-Head PPU, Hygon DCU and Apple Silicon nodes can share the same scheduler, and a built-in catalog provides launch settings for many popular models.

Beyond serving, the project offers GPU sharing with memory quotas, sleep/wake for taking turns on one card, multi-node parallelism, prefill/decode disaggregation, metrics, logs, request tracing, benchmarking and an MCP endpoint.

Key features

  • 01Deploys models to NVIDIA, AMD, Ascend, PPU, Hygon and Apple Silicon nodes in one cluster.
  • 02Estimates GPU memory needs and explains why each GPU was chosen or rejected.
  • 03Serves chat, embeddings, rerank, audio and image endpoints under an OpenAI-compatible /v1 API.
  • 04Syncs instances from several public clouds and schedules them with on-premises servers.
  • 05Lets GPU nodes connect outbound over WebSocket without opening inbound ports.
  • 06Provides GPU and engine metrics, searchable logs, request traces and a built-in benchmark.

Use cases

  • Running open-weight LLMs on a mix of rented cloud GPUs and on-premises servers.
  • Offering an internal OpenAI-compatible model endpoint for several teams with per-key usage tracking.
  • Hosting multiple small models on one GPU using memory quotas.
  • Operating inference in air-gapped sites with offline image and model import.

In collections

Self-hosted apps

Topics

#llm-serving#gpu-management#inference#openai-compatible#vllm#kubernetes-alternative#self-hosted#multi-cloud

FAQ

What is gpuworks?
A lightweight GPU management and Model-as-a-Service platform that pools servers and clouds with mixed accelerators behind one OpenAI-compatible endpoint. GPUWorks is a GPU management and Model-as-a-Service (MaaS) platform aimed at teams that run LLMs and other models on a mix of hardware. You join bare-metal servers, cloud instances, VMs or a Mac to one cluster, choose a model, and the platform estimates the GPU memory it needs, picks suitable GPUs, starts an inference engine such as vLLM or SGLang, and exposes it through a single OpenAI-compatible endpoint.
Is gpuworks free to use? What license does it have?
gpuworks is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is gpuworks still maintained?
Actively maintained. The last commit to gpuworks was on Nov 6, 2026.
What can I use gpuworks for?
Running open-weight LLMs on a mix of rented cloud GPUs and on-premises servers. Offering an internal OpenAI-compatible model endpoint for several teams with per-key usage tracking. Hosting multiple small models on one GPU using memory quotas. Operating inference in air-gapped sites with offline image and model import.
Where can I get gpuworks?
The source code and installation instructions are on GitHub: https://github.com/flitai-io/gpuworks

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects