Repolode
Submit
microsoft

microsoft/vattention

A KV-cache memory manager for LLM serving that allocates GPU memory on demand via CUDA virtual memory APIs, without PagedAttention.

View on GitHub
Stars
526
Forks
46
Open issues
13
Last commit
2 months ago

What is it?

vAttention is a memory manager for the KV-cache in LLM serving systems. It separates virtual memory allocation from physical memory allocation using the CUDA virtual memory APIs, so physical GPU memory is mapped on demand while the KV-cache stays contiguous in virtual address space.

Because the KV-cache remains contiguous, vAttention can support dynamic allocation with unmodified attention kernels. This contrasts with PagedAttention, which implements paging in user space and requires custom kernels. The project includes a vAttention allocator, a modified version of Sarathi-Serve that supports both PagedAttention and vAttention, benchmark scripts, and an OpenAI-compatible API server for benchmarking.

The repository targets Linux with NVIDIA GPUs (tested on A100), PyTorch 2.3.0, CUDA 12.1 or later, and Python 3.10. It supports FlashAttention and FlashInfer backends and page sizes from 64KB to 2MB, with smaller pages requiring a custom UVM driver.

Key features

  • 01Allocates physical GPU memory on demand while keeping KV-cache contiguous in virtual memory.
  • 02Works with unmodified attention kernels, unlike PagedAttention.
  • 03Supports FlashAttention and FlashInfer attention backends.
  • 04Offers page sizes from 64KB to 2MB, with asynchronous allocation overlapped with compute.
  • 05Includes benchmark runners and an OpenAI-compatible API server.

Use cases

  • Researching or prototyping KV-cache memory management for LLM inference.
  • Benchmarking serving throughput and latency on prefill-heavy or long-context workloads.
  • Integrating dynamic KV-cache allocation into a serving system without rewriting attention kernels.

Quick start

  1. 1

    Create and activate a conda environment with Python 3.10

    conda create -n vattn python=3.10
  2. 2

    Download and extract libtorch matching PyTorch 2.3.0

    wget https://download.pytorch.org/libtorch/cu121/libtorch-shared-with-deps-2.3.0%2Bcu121.zip
  3. 3

    Install Sarathi-Serve from the sarathi-lean directory

    pip install -e . --extra-index-url https://flashinfer.ai/whl/cu121/torch2.3/
  4. 4

    Build the vAttention allocator, setting LIBTORCH_PATH to your libtorch directory

    LIBTORCH_PATH=<path to libtorch dir> python setup.py install

Things to know

  • Tested mainly on Linux with A100 GPUs and CUDA 12.1 or later, so other hardware may not work.
  • Requires specific PyTorch 2.3.0 and matching libtorch versions.
  • Research prototype without full feature parity with open-source vLLM.

Topics

#llm-serving#kv-cache#cuda#gpu-memory#inference#pytorch#memory-management

FAQ

What is vattention?
A KV-cache memory manager for LLM serving that allocates GPU memory on demand via CUDA virtual memory APIs, without PagedAttention. vAttention is a memory manager for the KV-cache in LLM serving systems. It separates virtual memory allocation from physical memory allocation using the CUDA virtual memory APIs, so physical GPU memory is mapped on demand while the KV-cache stays contiguous in virtual address space.
Is vattention free to use? What license does it have?
vattention is open source under the MIT license. Check the license terms before using it in commercial products.
Is vattention still maintained?
Updated within 6 months. The last commit to vattention was on Aug 24, 2026.
What can I use vattention for?
Researching or prototyping KV-cache memory management for LLM inference. Benchmarking serving throughput and latency on prefill-heavy or long-context workloads. Integrating dynamic KV-cache allocation into a serving system without rewriting attention kernels.
How do I install vattention?
Create and activate a conda environment with Python 3.10: conda create -n vattn python=3.10 Download and extract libtorch matching PyTorch 2.3.0: wget https://download.pytorch.org/libtorch/cu121/libtorch-shared-with-deps-2.3.0%2Bcu121.zip Install Sarathi-Serve from the sarathi-lean directory: pip install -e . --extra-index-url https://flashinfer.ai/whl/cu121/torch2.3/ Build the vAttention allocator, setting LIBTORCH_PATH to your libtorch directory: LIBTORCH_PATH=<path to libtorch dir> python setup.py install
Where can I get vattention?
The source code and installation instructions are on GitHub: https://github.com/microsoft/vattention

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects