Repolode
Submit
microsoft

microsoft/MInference

A pip-installable library that speeds up pre-filling for long-context LLMs using dynamic sparse attention, with little loss in accuracy.

View on GitHubWebsite
Stars
1,232
Forks
85
Open issues
99
Last commit
29 days ago

What is it?

MInference is a library for accelerating the pre-filling stage of long-context LLM inference. Pre-filling, where the model processes the entire prompt before generating output, becomes expensive as prompts grow to hundreds of thousands or a million tokens. MInference approximates sparse attention patterns at runtime and computes attention with custom kernels, which the project reports can reduce pre-filling latency by up to 10x on an A100 while maintaining accuracy.

It integrates with Hugging Face Transformers and vLLM by patching a model with a few lines of code, and it also exposes its sparse attention kernels directly for use in custom pipelines. The project also includes SCBench, a benchmark that evaluates long-context methods across the KV cache lifecycle, and MMInference, an extension for accelerating long-context vision-language models.

Supported models include Qwen2.5, LLaMA-3 and 3.1, GLM-4, Yi, Phi-3 and Qwen2, and the library states it can work with any decoding LLM. It also supports several KV cache compression and retrieval methods such as SnapKV, StreamingLLM and Quest.

Key features

  • 01Dynamic sparse attention that approximates sparse indices online for faster pre-filling.
  • 02Integration with Hugging Face Transformers and vLLM through a simple model patch.
  • 03Direct access to sparse attention kernels for custom inference code.
  • 04Support for multiple KV cache methods, including SnapKV, StreamingLLM and Quest.
  • 05SCBench benchmark for evaluating long-context methods across the KV cache lifecycle.

Use cases

  • Serving long-context LLMs with prompts of hundreds of thousands of tokens on a single GPU.
  • Reducing time-to-first-token for document question answering or codebase analysis.
  • Researching sparse attention and KV cache methods for long-context models.

Quick start

  1. 1

    Install the library from PyPI with pip

    pip install minference
  2. 2

    Patch a Hugging Face model with MInference and run generation

Things to know

  • Requires Transformers 4.46.0 or newer and a GPU-oriented stack such as Torch and Triton.
  • Speedups are reported on long contexts and mainly on an A100 GPU, so benefits on short prompts or other hardware may be smaller.

In collections

Run LLMs locally

Topics

#llm-inference#sparse-attention#long-context#pytorch#vllm#transformers#kv-cache

FAQ

What is MInference?
A pip-installable library that speeds up pre-filling for long-context LLMs using dynamic sparse attention, with little loss in accuracy. MInference is a library for accelerating the pre-filling stage of long-context LLM inference. Pre-filling, where the model processes the entire prompt before generating output, becomes expensive as prompts grow to hundreds of thousands or a million tokens. MInference approximates sparse attention patterns at runtime and computes attention with custom kernels, which the project reports can reduce pre-filling latency by up to 10x on an A100 while maintaining accuracy.
Is MInference free to use? What license does it have?
MInference is open source under the MIT license. Check the license terms before using it in commercial products.
Is MInference still maintained?
Actively maintained. The last commit to MInference was on Oct 10, 2026.
What can I use MInference for?
Serving long-context LLMs with prompts of hundreds of thousands of tokens on a single GPU. Reducing time-to-first-token for document question answering or codebase analysis. Researching sparse attention and KV cache methods for long-context models.
How do I install MInference?
Install the library from PyPI with pip: pip install minference Patch a Hugging Face model with MInference and run generation
Where can I get MInference?
The source code and installation instructions are on GitHub: https://github.com/microsoft/MInference

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects