Repolode
Submit
runpod-workers

runpod-workers/worker-vllm

Runpod Serverless worker template that deploys OpenAI-compatible LLM endpoints powered by the vLLM inference engine.

View on GitHub
Stars
474
Forks
394
Open issues
25
Last commit
yesterday

What is it?

worker-vllm is a Docker-based worker for Runpod Serverless that runs large language models with the vLLM inference engine. You point it at a Hugging Face model repo or a baked-in local model, configure it through environment variables, and it exposes OpenAI-compatible Chat Completions, Models, Responses and Messages routes, with streaming and non-streaming support.

The worker translates environment variables such as MAX_MODEL_LEN, QUANTIZATION and TENSOR_PARALLEL_SIZE into vllm serve flags, and also accepts a YAML config file. Requests can also be sent through a native job format with prompt or messages inputs and sampling parameters. When a model cannot start because of causes like insufficient GPU memory or a missing HF_TOKEN, the worker stays running and returns the cause as a startup error in job responses.

The project provides a pre-built image, a Dockerfile option for baking a model into the image, and documentation for deploying through the Runpod console. It also notes that a separate load balancing setup can use the official vLLM image directly without this worker.

Key features

  • 01Serves OpenAI-compatible Chat Completions, Models, Responses and Messages routes with streaming support.
  • 02Maps environment variables to vllm serve flags, and accepts a YAML config file.
  • 03Loads models from Hugging Face repos or from a local path baked into the image.
  • 04Supports quantization options such as AWQ, GPTQ and bitsandbytes, plus multi-GPU tensor parallelism.
  • 05Reports startup failures like insufficient GPU memory as readable errors in job responses instead of crash-looping.

Use cases

  • Hosting an open-weight LLM on Runpod GPUs behind an OpenAI-compatible API.
  • Switching an existing OpenAI SDK application to a self-hosted model by changing the base URL and API key.
  • Running bursty inference workloads on serverless GPUs that scale to zero when idle.

Quick start

  1. 1

    Deploy the pre-built Docker image via the Runpod Console using the deploy guide

  2. 2

    Set MODEL_NAME to a Hugging Face repo ID or local model path in the environment variables

  3. 3

    Build a custom image with the model baked in (optional)

    docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg BASE_PATH="/models" .

Things to know

  • Requires a Runpod Serverless account and GPU resources to run the models.
  • Gated or private models need a HuggingFace token (HF_TOKEN).
  • Supported models are limited to those vLLM supports.

In collections

Run and train AI on GPUs

Topics

#llm#vllm#inference#serverless#openai-compatible#gpu#runpod#language-model

FAQ

What is worker-vllm?
Runpod Serverless worker template that deploys OpenAI-compatible LLM endpoints powered by the vLLM inference engine. worker-vllm is a Docker-based worker for Runpod Serverless that runs large language models with the vLLM inference engine. You point it at a Hugging Face model repo or a baked-in local model, configure it through environment variables, and it exposes OpenAI-compatible Chat Completions, Models, Responses and Messages routes, with streaming and non-streaming support.
Is worker-vllm free to use? What license does it have?
worker-vllm is open source under the MIT license. Check the license terms before using it in commercial products.
Is worker-vllm still maintained?
Actively maintained. The last commit to worker-vllm was on Nov 5, 2026.
What can I use worker-vllm for?
Hosting an open-weight LLM on Runpod GPUs behind an OpenAI-compatible API. Switching an existing OpenAI SDK application to a self-hosted model by changing the base URL and API key. Running bursty inference workloads on serverless GPUs that scale to zero when idle.
How do I install worker-vllm?
Deploy the pre-built Docker image via the Runpod Console using the deploy guide Set MODEL_NAME to a Hugging Face repo ID or local model path in the environment variables Build a custom image with the model baked in (optional): docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg BASE_PATH="/models" .
Where can I get worker-vllm?
The source code and installation instructions are on GitHub: https://github.com/runpod-workers/worker-vllm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects