A Python library from NVIDIA for compressing LLM KV caches during inference, with many pluggable compression methods built on Hugging Face transformers.
1.2K+4 this week196PythonApache-2.0Updated yesterday
Zefan-Cai/KVCache-Factory
A unified research playground for comparing KV cache compression, retrieval, merging and quantization methods for long-context LLM inference.
- Stars
- 1,394
- Forks
- 181
- Open issues
- 4
- Last commit
- 3 months ago
What is it?
KVCache-Factory collects multiple KV cache methods for auto-regressive LLMs behind one evaluation interface. It began as the PyramidKV codebase and now includes baselines such as StreamingLLM, H2O, SnapKV, Quest, MiniCache, AdaKV and HeadKV, plus a lossless HeadInfer offloading path that keeps the full cache.
The project targets long-context inference, where the KV cache grows with prompt length and dominates GPU memory. Users pick a method and a per-layer cache budget via command-line flags, then run LongBench, RULER or needle-in-a-haystack evaluations to compare accuracy. A latency and memory benchmark script is included for measuring decoding cost.
Support is strongest for Llama and Mistral models. Some newer methods have narrower runner coverage, and the project pins transformers to 4.44.2 because its attention patches depend on that version.
Key features
- 01Bundles compression, retrieval, merging and quantization methods under one interface.
- 02Runs LongBench, RULER and needle-in-a-haystack evaluations with a configurable cache budget.
- 03Includes HeadInfer, a lossless head-wise CPU offloading method that keeps the full cache.
- 04Provides a latency and peak memory benchmark script for decoding.
- 05Supports FlashAttention v2 and SDPA attention implementations.
Use cases
- Comparing KV cache compression methods on long-context benchmarks.
- Reproducing PyramidKV or related cache-budget experiments.
- Measuring the memory and latency trade-off of KV cache strategies on a single GPU.
- Prototyping a new KV cache method against existing baselines.
Quick start
- 1
Clone the repository
git clone https://github.com/Zefan-Cai/KVCache-Factory.git - 2
Enter the project directory
cd KVCache-Factory - 3
Install the dependencies
pip install -r requirements.txt - 4
Set the Python path to the project root
export PYTHONPATH="$PWD:${PYTHONPATH}"
Things to know
- Requires a local Llama or Mistral model and a GPU for the main evaluation runs.
- Pinned to transformers 4.44.2, and the attention patches are version-sensitive.
- Some newer methods have narrower runner and model coverage.
- Batch inference is not supported; eval_batch_size must stay at 1.
Topics
FAQ
- What is KVCache-Factory?
- A unified research playground for comparing KV cache compression, retrieval, merging and quantization methods for long-context LLM inference. KVCache-Factory collects multiple KV cache methods for auto-regressive LLMs behind one evaluation interface. It began as the PyramidKV codebase and now includes baselines such as StreamingLLM, H2O, SnapKV, Quest, MiniCache, AdaKV and HeadKV, plus a lossless HeadInfer offloading path that keeps the full cache.
- Is KVCache-Factory free to use? What license does it have?
- KVCache-Factory is open source under the MIT license. Check the license terms before using it in commercial products.
- Is KVCache-Factory still maintained?
- Updated within 6 months. The last commit to KVCache-Factory was on Aug 13, 2026.
- What can I use KVCache-Factory for?
- Comparing KV cache compression methods on long-context benchmarks. Reproducing PyramidKV or related cache-budget experiments. Measuring the memory and latency trade-off of KV cache strategies on a single GPU. Prototyping a new KV cache method against existing baselines.
- How do I install KVCache-Factory?
- Clone the repository: git clone https://github.com/Zefan-Cai/KVCache-Factory.git Enter the project directory: cd KVCache-Factory Install the dependencies: pip install -r requirements.txt Set the Python path to the project root: export PYTHONPATH="$PWD:${PYTHONPATH}"
- Where can I get KVCache-Factory?
- The source code and installation instructions are on GitHub: https://github.com/Zefan-Cai/KVCache-Factory
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
A Python library for compressing LLMs through quantization and pruning, producing checkpoints that load directly into vLLM for inference.
4K+20 this week729PythonApache-2.0Updated 15 hours ago
A toolkit for quantizing large language models (GPTQ, AWQ, GGUF, FP8, EXL3 and more) and serving them on NVIDIA, AMD, Intel, Apple and CPU hardware.
1.3K+6 this week214PythonOtherUpdated 4 hours ago
A Python framework for compressing neural networks via quantization, pruning and sparsity to speed up inference in OpenVINO, PyTorch and ONNX.
1.2K+3 this week314PythonApache-2.0Well maintainedUpdated yesterday
An edge-native MoE serving engine that runs large open-weight models like DeepSeek-V4 and Qwen3.6 on consumer GPUs and CPUs.
16.8K+1.9K this week1.7KPythonApache-2.0Updated 10 hours ago
NVIDIA's open-source library for compressing deep learning models via quantization, pruning, distillation, sparsity and speculative decoding to speed up inference.
AI & Machine Learning#model-compression#quantization#pruning#knowledge-distillation#inference-optimization
5.6K+76 this week803PythonApache-2.0Updated 3 hours ago