Repolode
Submit
InftyAI

InftyAI/llmaz

A Kubernetes-native inference platform for deploying and serving large language models with pluggable backends like vLLM, SGLang and llama.cpp.

View on GitHubWebsite
Stars
314
Forks
48
Open issues
53
Last commit
2 months ago

What is it?

llmaz is an inference platform for running large language models on Kubernetes. Users declare a model and a playground as Kubernetes resources, and the platform handles model loading and service exposure, so a working LLM endpoint can be created with minimal configuration.

It integrates with several inference backends, including vLLM, Text-Generation-Inference, SGLang, llama.cpp and TensorRT-LLM, and can load models from Hugging Face, ModelScope or object stores. It also supports serving one model across heterogeneous devices, multi-host distributed inference through LWS, and autoscaling with HPA and Karpenter.

The project is currently in alpha, so its APIs may change before a Beta release. Optional integrations include Envoy AI Gateway for token-based rate limiting and model routing, and Open WebUI as a built-in chat interface.

Key features

  • 01Deploy an LLM service on Kubernetes by applying a model and a playground resource.
  • 02Supports vLLM, TGI, SGLang, llama.cpp and TensorRT-LLM backends.
  • 03Loads models automatically from Hugging Face, ModelScope and object stores.
  • 04Multi-host distributed inference with LWS.
  • 05Autoscaling with HPA on LLM-based metrics and node autoscaling via Karpenter.
  • 06Token-based rate limiting and model routing through Envoy AI Gateway.

Use cases

  • Hosting open-weight LLMs on an existing Kubernetes cluster with GPUs.
  • Serving one model across a mix of heterogeneous accelerators to balance cost and performance.
  • Running large models that need multi-host distributed inference.
  • Providing an OpenAI-compatible completions endpoint backed by a chat UI for internal teams.

Quick start

  1. 1

    Read the installation guide and install llmaz on your Kubernetes cluster

  2. 2

    Create a Hugging Face token secret if your model needs one

    kubectl create secret generic modelhub-secret --from-literal=HF_TOKEN=<your token>
  3. 3

    Apply the OpenModel and Playground resources to deploy facebook/opt-125m

  4. 4

    Port-forward the service to localhost

    kubectl port-forward svc/opt-125m-lb 8080:8080

Things to know

  • The project is in alpha, so its API may change before it reaches Beta.
  • Deployment requires an existing Kubernetes cluster, adding self-hosting effort.
  • Running on GPUs or CPUs requires additional setup, such as the llama.cpp example for CPU use.

In collections

Run LLMs locally

Topics

#llm#kubernetes#inference#model-serving#vllm#llama-cpp#distributed-inference#autoscaling#huggingface#inference-platform

FAQ

What is llmaz?
A Kubernetes-native inference platform for deploying and serving large language models with pluggable backends like vLLM, SGLang and llama.cpp. llmaz is an inference platform for running large language models on Kubernetes. Users declare a model and a playground as Kubernetes resources, and the platform handles model loading and service exposure, so a working LLM endpoint can be created with minimal configuration.
Is llmaz free to use? What license does it have?
llmaz is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is llmaz still maintained?
Updated within 6 months. The last commit to llmaz was on Aug 27, 2026.
What can I use llmaz for?
Hosting open-weight LLMs on an existing Kubernetes cluster with GPUs. Serving one model across a mix of heterogeneous accelerators to balance cost and performance. Running large models that need multi-host distributed inference. Providing an OpenAI-compatible completions endpoint backed by a chat UI for internal teams.
How do I install llmaz?
Read the installation guide and install llmaz on your Kubernetes cluster Create a Hugging Face token secret if your model needs one: kubectl create secret generic modelhub-secret --from-literal=HF_TOKEN=<your token> Apply the OpenModel and Playground resources to deploy facebook/opt-125m Port-forward the service to localhost: kubectl port-forward svc/opt-125m-lb 8080:8080
Where can I get llmaz?
The source code and installation instructions are on GitHub: https://github.com/InftyAI/llmaz

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects