Repolode
Submit
ome-projects

ome-projects/ome

A Kubernetes operator for deploying and managing LLM serving, with GPU scheduling and model lifecycle handling for SGLang, vLLM, TensorRT-LLM and Triton.

View on GitHubWebsite
Stars
518
Forks
104
Open issues
120
Last commit
17 hours ago

What is it?

OME (Open Model Engine) is a Kubernetes operator for deploying and operating large language models. It models base models, LoRA fine-tuned weights, serving runtimes and inference services as custom resources, and its controller downloads and parses model files, selects a matching runtime configuration, and generates the Kubernetes resources needed to run them.

The project targets clusters running multiple models on GPUs. It includes GPU bin-packing scheduling through a separate scheduler, AcceleratorClass definitions for hardware selection, prefill-decode disaggregation and multi-node inference, and canary or blue-green rollouts. Integrations include Kueue, LeaderWorkerSet, KEDA and the Gateway API Inference Extension.

A `kubectl ome` plugin inspects models, runtimes and services, streams logs, and submits guarded actions such as rollout rollback or traffic drain. Optional alpha add-ons cover fleet quota management and a GPU caretaker component. Installation is via Helm charts, and the project requires Kubernetes 1.28 or newer.

Key features

  • 01Models, runtimes and inference services are defined as Kubernetes custom resources.
  • 02Automatic runtime selection based on model architecture, format, quantization and size.
  • 03Supports SGLang and vLLM, with TensorRT-LLM and Triton also named in the project description.
  • 04Prefill-decode disaggregation, multi-node inference, and canary or blue-green rollouts.
  • 05GPU bin-packing scheduling and AcceleratorClass-based hardware selection.
  • 06A kubectl plugin for inspecting services and running guarded rollout and traffic actions.

Use cases

  • Serving several open-weight LLMs such as Llama, Qwen or DeepSeek on a shared GPU Kubernetes cluster.
  • Running large models across multiple nodes with prefill-decode disaggregation.
  • Managing LLM rollouts and autoscaling with KEDA, Kueue or the Gateway API.
  • Benchmarking model performance under configurable traffic patterns with BenchmarkJob resources.

Quick start

  1. 1

    Install the OME CRDs with Helm

    helm upgrade --install ome-crd oci://ghcr.io/moirai-internal/charts/ome-crd --namespace ome --create-namespace
  2. 2

    Install the OME resources with Helm

    helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources --namespace ome

Things to know

  • Requires Kubernetes 1.28 or newer and a Kubernetes cluster to run on.
  • Some add-ons such as the scheduler, quota manager and Alfred are alpha.

In collections

Run LLMs locally

Topics

#llm-inference#kubernetes#model-serving#gpu-scheduling#sglang#vllm#llm-ops#deepseek#k8s#kimi-k2

FAQ

What is ome?
A Kubernetes operator for deploying and managing LLM serving, with GPU scheduling and model lifecycle handling for SGLang, vLLM, TensorRT-LLM and Triton. OME (Open Model Engine) is a Kubernetes operator for deploying and operating large language models. It models base models, LoRA fine-tuned weights, serving runtimes and inference services as custom resources, and its controller downloads and parses model files, selects a matching runtime configuration, and generates the Kubernetes resources needed to run them.
Is ome free to use? What license does it have?
ome is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is ome still maintained?
Actively maintained. The last commit to ome was on Oct 22, 2026.
What can I use ome for?
Serving several open-weight LLMs such as Llama, Qwen or DeepSeek on a shared GPU Kubernetes cluster. Running large models across multiple nodes with prefill-decode disaggregation. Managing LLM rollouts and autoscaling with KEDA, Kueue or the Gateway API. Benchmarking model performance under configurable traffic patterns with BenchmarkJob resources.
How do I install ome?
Install the OME CRDs with Helm: helm upgrade --install ome-crd oci://ghcr.io/moirai-internal/charts/ome-crd --namespace ome --create-namespace Install the OME resources with Helm: helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources --namespace ome
Where can I get ome?
The source code and installation instructions are on GitHub: https://github.com/ome-projects/ome

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects