Repolode
Submit
kubeai-project

kubeai-project/kubeai

A Kubernetes operator that deploys and autoscales LLM, embedding, reranking and speech-to-text models behind an OpenAI-compatible API.

View on GitHubWebsite
Stars
1,289
Forks
144
Open issues
97
Last commit
4 days ago

What is it?

KubeAI is an AI inference operator for Kubernetes that deploys and scales machine learning models such as LLMs, embedding models, rerankers and speech-to-text models. It manages backend servers like vLLM, Ollama, FasterWhisper and Infinity as Kubernetes Pods, handling model downloads, volume mounting and dynamic LoRA adapter loading through a Model custom resource.

The project includes a proxy that exposes OpenAI-compatible endpoints such as chat completions, embeddings, rerank and audio transcriptions, so existing OpenAI client libraries can be pointed at it. The proxy uses prefix-aware load balancing to improve KV cache utilization across vLLM replicas, queues requests while a model scales up from zero replicas, and retries requests on failing backends.

KubeAI is installed with Helm and is designed to run without dependencies such as Istio, Knative or a Prometheus metrics adapter. It ships with a catalog of predefined models and supports CPU, GPU and TPU hardware.

Key features

  • 01Deploys vLLM, Ollama, FasterWhisper and Infinity servers as Kubernetes Pods.
  • 02Scales model replicas from zero based on demand.
  • 03Exposes OpenAI-compatible endpoints for chat, embeddings, rerank and transcription.
  • 04Prefix-aware load balancing to improve vLLM KV cache utilization.
  • 05Automates model downloading and volume mounting, and manages LoRA adapters.

Use cases

  • Hosting open-weight LLMs on a company's own Kubernetes cluster behind an OpenAI-style API.
  • Running speech-to-text transcription services that scale down when idle.
  • Serving embedding and reranking models for search or retrieval pipelines.

Quick start

  1. 1

    Create a local Kubernetes cluster

    kind create cluster
  2. 2

    Add the KubeAI Helm repository

    helm repo add kubeai https://www.kubeai.org
  3. 3

    Install KubeAI and wait for components to be ready

    helm install kubeai kubeai/kubeai --wait --timeout 10m
  4. 4

    Install predefined models using a values file (see README for the YAML)

Things to know

  • Requires a running Kubernetes cluster; it is not a standalone tool.
  • Local setup via Podman for kind needs at least 6 GB of memory for the machine.

In collections

Run LLMs locallyAI video & audio tools

Topics

#kubernetes#llm#inference#autoscaling#openai-api#vllm#ollama#whisper#ai#autoscaler

FAQ

What is kubeai?
A Kubernetes operator that deploys and autoscales LLM, embedding, reranking and speech-to-text models behind an OpenAI-compatible API. KubeAI is an AI inference operator for Kubernetes that deploys and scales machine learning models such as LLMs, embedding models, rerankers and speech-to-text models. It manages backend servers like vLLM, Ollama, FasterWhisper and Infinity as Kubernetes Pods, handling model downloads, volume mounting and dynamic LoRA adapter loading through a Model custom resource.
Is kubeai free to use? What license does it have?
kubeai is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is kubeai still maintained?
Actively maintained. The last commit to kubeai was on Nov 3, 2026.
What can I use kubeai for?
Hosting open-weight LLMs on a company's own Kubernetes cluster behind an OpenAI-style API. Running speech-to-text transcription services that scale down when idle. Serving embedding and reranking models for search or retrieval pipelines.
How do I install kubeai?
Create a local Kubernetes cluster: kind create cluster Add the KubeAI Helm repository: helm repo add kubeai https://www.kubeai.org Install KubeAI and wait for components to be ready: helm install kubeai kubeai/kubeai --wait --timeout 10m Install predefined models using a values file (see README for the YAML)
Where can I get kubeai?
The source code and installation instructions are on GitHub: https://github.com/kubeai-project/kubeai

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects