Repolode
Submit
flashinfer-ai

flashinfer-ai/flashinfer

A GPU kernel library and generator for LLM inference, providing unified APIs for attention, GEMM, MoE, and sampling operations.

View on GitHubWebsite
Stars
6,771
Forks
1,644
Open issues
1,276
Last commit
12 hours ago

What is it?

FlashInfer is a kernel library for LLM inference serving. It offers unified APIs for attention, GEMM, mixture-of-experts, sampling, and related operators, and can dispatch to multiple backend implementations such as FlashAttention-2/3, cuDNN, CUTLASS, and TensorRT-LLM depending on hardware and workload.

The library covers attention over paged and ragged KV-caches, including decode, prefill, and append phases, MLA attention for DeepSeek models, cascade attention for shared prefixes, and fused prefill-plus-decode kernels for mixed batching. It also includes FP8 and FP4 GEMM and quantized MoE kernels, sorting-free Top-K, Top-P, and Min-P sampling, and speculative decoding support.

Supported GPUs range from Turing (SM 7.5) through Blackwell (SM 12.x). Kernels can be compiled just-in-time on first use or installed as pre-compiled wheels, and the project lists SGLang, vLLM, and TensorRT-LLM among the inference engines that use it.

Key features

  • 01Attention kernels for paged KV-cache, prefill, decode, and MLA
  • 02FP8 and FP4 GEMM and quantized fused MoE kernels
  • 03Sorting-free Top-K, Top-P, and Min-P sampling
  • 04Speculative decoding and AllReduce/NVLink communication primitives
  • 05CUDAGraph and torch.compile compatible

Use cases

  • Building or tuning an LLM serving engine that needs optimized attention and MoE kernels
  • Running low-precision (FP8/FP4) inference on Hopper or Blackwell GPUs
  • Adding custom attention variants through JIT-compiled kernels
  • Benchmarking attention backends across different GPU generations

Quick start

  1. 1

    Install the core package from PyPI

    pip install flashinfer-python
  2. 2

    Verify the installation and view configuration

    flashinfer show-config

Things to know

  • Requires an NVIDIA GPU with a supported compute capability (SM 7.5 and later).
  • Not all features are supported across all compute capabilities.
  • Kernels compile or download on first use unless optional pre-compiled packages are installed.

In collections

Run LLMs locallyRun and train AI on GPUs

Topics

#llm-inference#cuda#gpu#attention#moe#kernels#pytorch#distributed-inference#jit#large-large-models

FAQ

What is flashinfer?
A GPU kernel library and generator for LLM inference, providing unified APIs for attention, GEMM, MoE, and sampling operations. FlashInfer is a kernel library for LLM inference serving. It offers unified APIs for attention, GEMM, mixture-of-experts, sampling, and related operators, and can dispatch to multiple backend implementations such as FlashAttention-2/3, cuDNN, CUTLASS, and TensorRT-LLM depending on hardware and workload.
Is flashinfer free to use? What license does it have?
flashinfer is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is flashinfer still maintained?
Actively maintained. The last commit to flashinfer was on Oct 24, 2026.
What can I use flashinfer for?
Building or tuning an LLM serving engine that needs optimized attention and MoE kernels Running low-precision (FP8/FP4) inference on Hopper or Blackwell GPUs Adding custom attention variants through JIT-compiled kernels Benchmarking attention backends across different GPU generations
How do I install flashinfer?
Install the core package from PyPI: pip install flashinfer-python Verify the installation and view configuration: flashinfer show-config
Where can I get flashinfer?
The source code and installation instructions are on GitHub: https://github.com/flashinfer-ai/flashinfer

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects