Repolode
Submit
kaito-project

kaito-project/kaito

A Kubernetes operator suite that automates deploying LLM inference, fine-tuning, and RAG services onto GPU nodes.

View on GitHubWebsite
Stars
1,037
Forks
195
Open issues
73
Last commit
yesterday

What is it?

KAITO is an operator suite for Kubernetes that automates running large language model inference, fine-tuning, and Retrieval Augmented Generation (RAG) engines in a cluster. Users supply a GPU instance type and a HuggingFace model ID through a Workspace resource, and the controller estimates the GPU memory needed, calculates the GPU count, and triggers node provisioning through Karpenter NodePool APIs.

The controller also generates inference engine settings for single-node or multi-node serving based on GPU topology, so users avoid tuning parallelism and max model length by hand. Currently only the vLLM engine is supported, with LoRA adapters and KV cache offloading. InferenceSet handles replica scaling driven by vLLM metrics via KEDA, and InferencePool integrates with the Gateway API Inference Extension for KV-cache-aware routing.

A separate RAGEngine resource deploys an LLM endpoint, embedding service, and vector database (FAISS by default, or Qdrant and Milvus) together, with a RAGService built on LlamaIndex that supports document indexing and hybrid BM25 plus vector retrieval.

Key features

  • 01Workspace CRD that estimates GPU memory and count from a HuggingFace model ID and instance type.
  • 02Automatic GPU node provisioning via Karpenter NodePool APIs.
  • 03Preset inference parameters for single-node and multi-node vLLM serving.
  • 04KEDA-based autoscaling of inference replicas using vLLM metrics.
  • 05RAGEngine with FAISS, Qdrant, or Milvus vector stores and hybrid BM25 plus vector retrieval.

Use cases

  • Serving open-weight LLMs from HuggingFace on a Kubernetes cluster with GPU nodes.
  • Deploying a RAG service with document indexing and an OpenAI-style chat endpoint.
  • Autoscaling LLM inference replicas based on request load.
  • Running multi-node distributed inference for models that exceed a single GPU node.

In collections

Run LLMs locallyRun and train AI on GPUs

Topics

#kubernetes#llm-inference#gpu#operator#rag#vllm#autoscaling#ai

FAQ

What is kaito?
A Kubernetes operator suite that automates deploying LLM inference, fine-tuning, and RAG services onto GPU nodes. KAITO is an operator suite for Kubernetes that automates running large language model inference, fine-tuning, and Retrieval Augmented Generation (RAG) engines in a cluster. Users supply a GPU instance type and a HuggingFace model ID through a Workspace resource, and the controller estimates the GPU memory needed, calculates the GPU count, and triggers node provisioning through Karpenter NodePool APIs.
Is kaito free to use? What license does it have?
kaito is open source under the Other license. Check the license terms before using it in commercial products.
Is kaito still maintained?
Actively maintained. The last commit to kaito was on Nov 6, 2026.
What can I use kaito for?
Serving open-weight LLMs from HuggingFace on a Kubernetes cluster with GPU nodes. Deploying a RAG service with document indexing and an OpenAI-style chat endpoint. Autoscaling LLM inference replicas based on request load. Running multi-node distributed inference for models that exceed a single GPU node.
Where can I get kaito?
The source code and installation instructions are on GitHub: https://github.com/kaito-project/kaito

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects