Repolode
Submit
defilantech

defilantech/LLMKube

Kubernetes operator for running self-hosted LLMs across NVIDIA, AMD and Apple Silicon hardware with one set of CRDs and OpenAI-compatible endpoints.

View on GitHubWebsite
Stars
251
Forks
43
Open issues
107
Last commit
20 hours ago

What is it?

LLMKube is a Kubernetes operator for deploying and managing self-hosted LLM inference. You define a Model and an InferenceService in YAML, and the operator handles model download and caching, GPU scheduling, health checks, scaling, and exposing an OpenAI-compatible API. It supports llama.cpp, vLLM and TGI runtimes inside the cluster.

Its distinguishing feature is running inference on Apple Silicon. A Metal Agent runs llama-server, mlx-server or vllm-swift as native macOS processes with Metal GPU access, and the controller exposes these as ordinary Kubernetes Services. This lets one cluster combine NVIDIA, AMD (Vulkan), Intel and Mac hosts under the same CRDs.

A ModelRouter resource places a policy-aware OpenAI-compatible endpoint in front of local InferenceServices and external providers such as Anthropic, OpenAI and LiteLLM. Rules can keep PII-classified requests local and return HTTP 503 instead of falling back to cloud backends. The project also includes Foreman, an opt-in orchestrator for running coding agents on the same fleet.

Key features

  • 01Kubernetes CRDs (Model, InferenceService) for declarative LLM deployment
  • 02Runtimes include llama.cpp, vLLM and TGI in-cluster, plus native llama-server, mlx-server and vllm-swift on Apple Silicon
  • 03Supports NVIDIA CUDA, AMD via Vulkan, Intel GPUs and Apple Metal in one fleet
  • 04OpenAI-compatible /v1/chat/completions endpoint with streaming
  • 05ModelRouter for policy-based routing between local and cloud models with fail-closed rules
  • 06Prometheus metrics, OpenTelemetry tracing and Grafana dashboards included

Use cases

  • Running LLM inference on-prem or in air-gapped environments under Kubernetes
  • Combining Mac Studio hosts and NVIDIA servers into one managed inference cluster
  • Routing regulated data to local models while sending other requests to cloud providers
  • Homelab setups that want multi-GPU or mixed-hardware model serving

Quick start

  1. 1

    Install the CLI with Homebrew

    brew install defilantech/tap/llmkube
  2. 2

    Install the operator on your Kubernetes cluster with Helm

    helm repo add llmkube https://defilantech.github.io/LLMKube
  3. 3

    Deploy a model with the CLI

    llmkube deploy phi-4-mini
  4. 4

    Port-forward the service and query the OpenAI-compatible endpoint

    kubectl port-forward svc/phi-4-mini 8080:8080 &

Things to know

  • Requires a Kubernetes cluster, so it is overkill for running one model on one machine.
  • Apple Silicon support needs the separate metal-agent running on a Mac.

In collections

Self-hosted appsRun LLMs locallyOpen-source Mac appsRun and train AI on GPUs

Topics

#llm-inference#kubernetes-operator#self-hosted#apple-silicon#gpu#llama-cpp#openai-compatible#local-llm#ai#autoscaling

FAQ

What is LLMKube?
Kubernetes operator for running self-hosted LLMs across NVIDIA, AMD and Apple Silicon hardware with one set of CRDs and OpenAI-compatible endpoints. LLMKube is a Kubernetes operator for deploying and managing self-hosted LLM inference. You define a Model and an InferenceService in YAML, and the operator handles model download and caching, GPU scheduling, health checks, scaling, and exposing an OpenAI-compatible API. It supports llama.cpp, vLLM and TGI runtimes inside the cluster.
Is LLMKube free to use? What license does it have?
LLMKube is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is LLMKube still maintained?
Actively maintained. The last commit to LLMKube was on Nov 6, 2026.
What can I use LLMKube for?
Running LLM inference on-prem or in air-gapped environments under Kubernetes Combining Mac Studio hosts and NVIDIA servers into one managed inference cluster Routing regulated data to local models while sending other requests to cloud providers Homelab setups that want multi-GPU or mixed-hardware model serving
How do I install LLMKube?
Install the CLI with Homebrew: brew install defilantech/tap/llmkube Install the operator on your Kubernetes cluster with Helm: helm repo add llmkube https://defilantech.github.io/LLMKube Deploy a model with the CLI: llmkube deploy phi-4-mini Port-forward the service and query the OpenAI-compatible endpoint: kubectl port-forward svc/phi-4-mini 8080:8080 &
Where can I get LLMKube?
The source code and installation instructions are on GitHub: https://github.com/defilantech/LLMKube

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects