Repolode
Submit
NVIDIA

NVIDIA/kvpress

A Python library from NVIDIA for compressing LLM KV caches during inference, with many pluggable compression methods built on Hugging Face transformers.

View on GitHub
Stars
1,245
Forks
194
Open issues
12
Last commit
yesterday

What is it?

Long-context LLM deployments are expensive because the key-value (KV) cache grows linearly with the number of tokens. kvpress addresses this by implementing a range of KV cache compression methods, called presses, and benchmarking them on top of Hugging Face transformers. Each press prunes or transforms cached entries during the prefilling phase and exposes a compression_ratio that sets how much of the cache is kept.

The library registers a custom text generation pipeline named kv-press-text-generation, which handles chat templates and tokenization. Compression can be applied to the context only, so the same compressed context can be queried with different questions. A DecodingPress wrapper, marked experimental, compresses the cache periodically during token generation using a target size instead of a ratio. Quantized caches via transformers' QuantizedCache are also supported.

The repository includes a large set of presses, from simple baselines such as random and key-norm scoring to methods like SnapKV, StreamingLLM, KVzip, and CriticalKV, plus wrapper presses such as AdaKVPress for head-wise compression. It also ships an evaluation CLI for long-context benchmarks and notebooks for measuring memory and speed gains.

Key features

  • 01Multiple KV cache compression methods (presses) with a shared compression_ratio interface
  • 02Custom transformers pipeline that compresses only the context and handles chat templates
  • 03Experimental decoding-time compression with a target cache size
  • 04Wrapper presses that combine or refine other scorers, such as head-wise AdaKV
  • 05Command-line evaluation on long-context benchmarks and notebooks for memory and speed measurement

Use cases

  • Answering many questions over one long document without re-encoding the context each time
  • Researching or prototyping new KV cache compression algorithms against common baselines
  • Reducing GPU memory for long-context inference on Llama, Qwen, Mistral and similar models

Quick start

  1. 1

    Install from PyPI

    pip install kvpress
  2. 2

    Load the pipeline and compress a context with a press

Things to know

  • Requires Hugging Face transformers and a model loadable through it, which typically needs a GPU for large models.
  • Most presses are research methods and some, like BlockPress, are not true chunked-prefill implementations.

Topics

#llm#kv-cache#kv-cache-compression#long-context#inference#pytorch#transformers#large-language-models#python

FAQ

What is kvpress?
A Python library from NVIDIA for compressing LLM KV caches during inference, with many pluggable compression methods built on Hugging Face transformers. Long-context LLM deployments are expensive because the key-value (KV) cache grows linearly with the number of tokens. kvpress addresses this by implementing a range of KV cache compression methods, called presses, and benchmarking them on top of Hugging Face transformers. Each press prunes or transforms cached entries during the prefilling phase and exposes a compression_ratio that sets how much of the cache is kept.
Is kvpress free to use? What license does it have?
kvpress is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is kvpress still maintained?
Actively maintained. The last commit to kvpress was on Nov 6, 2026.
What can I use kvpress for?
Answering many questions over one long document without re-encoding the context each time Researching or prototyping new KV cache compression algorithms against common baselines Reducing GPU memory for long-context inference on Llama, Qwen, Mistral and similar models
How do I install kvpress?
Install from PyPI: pip install kvpress Load the pipeline and compress a context with a press
Where can I get kvpress?
The source code and installation instructions are on GitHub: https://github.com/NVIDIA/kvpress

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects