Repolode
Submit
vllm-project

vllm-project/vllm

A fast, memory-efficient library and server for running and serving large language models with an OpenAI-compatible API.

View on GitHubWebsite
Stars
94,202
Forks
23,633
Open issues
9,134
Last commit
6 hours ago

What is it?

vLLM is a library for LLM inference and serving. It manages attention key-value memory with PagedAttention and batches incoming requests continuously, aiming for high throughput at low cost. It loads many Hugging Face model architectures, including dense, mixture-of-experts, multimodal, and embedding models.

The engine supports a wide range of quantization formats such as FP8, INT4, GPTQ, AWQ, and GGUF, along with speculative decoding, prefix caching, and multi-LoRA serving. It can be distributed across GPUs with tensor, pipeline, data, expert, and context parallelism.

vLLM runs on NVIDIA and AMD GPUs, Intel GPUs, and x86, ARM, and PowerPC CPUs, with additional hardware plugins for TPUs, Intel Gaudi, Huawei Ascend, and Apple Silicon. It exposes an OpenAI-compatible API server, and can be installed with uv or pip.

Key features

  • 01PagedAttention for efficient KV cache memory management
  • 02Continuous batching, chunked prefill, and prefix caching
  • 03Broad quantization support including FP8, INT4, GPTQ, AWQ, and GGUF
  • 04OpenAI-compatible API server with structured outputs and tool calling
  • 05Tensor, pipeline, expert, and context parallelism for multi-GPU inference
  • 06Support for NVIDIA, AMD, Intel GPUs and CPUs, plus TPU and other hardware plugins

Use cases

  • Serving open-weight LLMs like Llama or Qwen behind an OpenAI-compatible endpoint
  • Running high-throughput batch inference on a multi-GPU server
  • Deploying quantized MoE models such as DeepSeek-V3 or GPT-OSS
  • Serving multiple LoRA fine-tunes from one base model

Quick start

  1. 1

    Install vLLM with uv (recommended)

    uv pip install vllm
  2. 2

    Download precompiled kernels for faster startup

    vllm download-kernels

Things to know

  • Needs a GPU or supported accelerator to get the full performance benefits.
  • Large model weights and dependencies make setup heavier than lightweight tools.

Topics

#llm-serving#inference#pytorch#cuda#openai-api#quantization#gpu#amd#blackwell#deepseek

FAQ

What is vllm?
A fast, memory-efficient library and server for running and serving large language models with an OpenAI-compatible API. vLLM is a library for LLM inference and serving. It manages attention key-value memory with PagedAttention and batches incoming requests continuously, aiming for high throughput at low cost. It loads many Hugging Face model architectures, including dense, mixture-of-experts, multimodal, and embedding models.
Is vllm free to use? What license does it have?
vllm is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is vllm still maintained?
Actively maintained. The last commit to vllm was on Oct 20, 2026.
What can I use vllm for?
Serving open-weight LLMs like Llama or Qwen behind an OpenAI-compatible endpoint Running high-throughput batch inference on a multi-GPU server Deploying quantized MoE models such as DeepSeek-V3 or GPT-OSS Serving multiple LoRA fine-tunes from one base model
How do I install vllm?
Install vLLM with uv (recommended): uv pip install vllm Download precompiled kernels for faster startup: vllm download-kernels
Where can I get vllm?
The source code and installation instructions are on GitHub: https://github.com/vllm-project/vllm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects