Repolode
Submit
vllm-project

vllm-project/vllm-omni

A serving framework that extends vLLM to run omni-modality models producing text, image, audio, video and action outputs.

View on GitHubWebsite
Stars
7,255
Forks
1,934
Open issues
1,900
Last commit
yesterday

What is it?

vLLM-Omni extends the vLLM inference engine beyond autoregressive text generation. It is built to serve models that take and produce text, image, audio, video and action data, including Diffusion Transformers and other parallel generation models alongside standard LLM-style decoding.

The framework uses a heterogeneous pipeline abstraction that splits a model workflow into stages, with stage execution overlapped for throughput and resources allocated dynamically across stages through its OmniConnector. It works with many Hugging Face models, including omni models such as Qwen3-Omni, TTS models such as Qwen3-TTS and CosyVoice3, and diffusion models such as Wan2.2 and LTX-2.5.

It provides an OpenAI-compatible API server, streaming outputs, full-duplex realtime serving with streaming audio input and output, and tensor, pipeline, data and expert parallelism for distributed inference.

Key features

  • 01Serves text, image, audio, video and action generation models in one runtime.
  • 02Runs Diffusion Transformers and other non-autoregressive models alongside vLLM's AR support.
  • 03Splits model workflows into pipelined stages with dynamic resource allocation.
  • 04Provides an OpenAI-compatible API server with streaming outputs.
  • 05Supports full-duplex realtime serving with streaming audio input and output.
  • 06Offers tensor, pipeline, data and expert parallelism for distributed inference.

Use cases

  • Serving a speech-to-speech or TTS model behind an OpenAI-style API.
  • Deploying diffusion image or video generation models at production scale.
  • Running multi-stage omni-modality pipelines that mix LLM and diffusion components.
  • Hosting robot-policy or action models with the same serving stack as LLMs.

Quick start

  1. 1

    Read the installation guide in the documentation

  2. 2

    Follow the quickstart guide in the documentation

  3. 3

    Optionally pull the nightly CUDA Docker image from Docker Hub

Things to know

  • The README gives no direct install command, so setup depends on the external installation and quickstart docs.
  • Releases are tied to specific upstream vLLM versions, so the project depends on vLLM's release line.
  • Many supported models are large diffusion or omni-modality models that typically need accelerator hardware such as CUDA, ROCm, XPU or NPU GPUs.

Topics

#inference#model-serving#multimodal#diffusion#image-generation#audio-generation#video-generation#pytorch#transformer#world-model

FAQ

What is vllm-omni?
A serving framework that extends vLLM to run omni-modality models producing text, image, audio, video and action outputs. vLLM-Omni extends the vLLM inference engine beyond autoregressive text generation. It is built to serve models that take and produce text, image, audio, video and action data, including Diffusion Transformers and other parallel generation models alongside standard LLM-style decoding.
Is vllm-omni free to use? What license does it have?
vllm-omni is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is vllm-omni still maintained?
Actively maintained. The last commit to vllm-omni was on Oct 19, 2026.
What can I use vllm-omni for?
Serving a speech-to-speech or TTS model behind an OpenAI-style API. Deploying diffusion image or video generation models at production scale. Running multi-stage omni-modality pipelines that mix LLM and diffusion components. Hosting robot-policy or action models with the same serving stack as LLMs.
How do I install vllm-omni?
Read the installation guide in the documentation Follow the quickstart guide in the documentation Optionally pull the nightly CUDA Docker image from Docker Hub
Where can I get vllm-omni?
The source code and installation instructions are on GitHub: https://github.com/vllm-project/vllm-omni

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects