Repolode
Submit
Dao-AILab

Dao-AILab/flash-attention

Official implementation of FlashAttention, a memory-efficient exact attention kernel for training and inference of Transformer models on GPUs.

View on GitHub
Stars
25,163
Forks
3,175
Open issues
1,382
Last commit
11 hours ago

What is it?

FlashAttention is an exact attention implementation for Transformer models that reduces GPU memory use and speeds up the attention computation. Standard attention needs memory that grows quadratically with sequence length; FlashAttention computes the same result with memory that grows linearly, which allows much longer sequences to fit on a GPU.

The repository contains FlashAttention-2 for NVIDIA CUDA (Ampere, Ada and Hopper GPUs) and AMD ROCm, plus FlashAttention-3 for Hopper GPUs and FlashAttention-4 written in CuTeDSL for Hopper and Blackwell GPUs. The main Python functions cover standard, variable-length, packed QKV, and KV-cache inference attention, with support for causal masks, sliding window attention, ALiBi, MQA/GQA, dropout and softcapping.

The package is installed with pip, and it requires PyTorch and a CUDA or ROCm toolkit. Linux is the main supported platform.

Key features

  • 01Computes exact attention with memory that scales linearly in sequence length.
  • 02Supports causal masking, sliding window local attention and ALiBi.
  • 03Provides MQA and GQA through KV tensors with fewer heads than Q.
  • 04Includes a paged KV cache and an in-place KV cache update for incremental decoding.
  • 05Offers CUDA builds for NVIDIA GPUs and Triton and CK backends for AMD GPUs.

Use cases

  • Training large Transformer language models with long sequences on NVIDIA A100 or H100 GPUs.
  • Serving LLM inference with incremental decoding and a paged KV cache.
  • Running attention on AMD Instinct GPUs through the ROCm backends.

Quick start

  1. 1

    Install the build prerequisites: packaging, psutil and ninja

  2. 2

    Install FlashAttention from PyPI

    pip install flash-attn --no-build-isolation

Things to know

  • Officially supported on Linux only; Windows compilation is not well tested.
  • Needs a CUDA or ROCm toolkit and a supported GPU, such as Ampere, Ada or Hopper for CUDA.
  • Compiling from source can be slow and memory-hungry without ninja and limited parallel jobs.

Topics

#attention#transformer#cuda#gpu#pytorch#deep-learning#llm#rocm

FAQ

What is flash-attention?
Official implementation of FlashAttention, a memory-efficient exact attention kernel for training and inference of Transformer models on GPUs. FlashAttention is an exact attention implementation for Transformer models that reduces GPU memory use and speeds up the attention computation. Standard attention needs memory that grows quadratically with sequence length; FlashAttention computes the same result with memory that grows linearly, which allows much longer sequences to fit on a GPU.
Is flash-attention free to use? What license does it have?
flash-attention is open source under the BSD-3-Clause license. Check the license terms before using it in commercial products.
Is flash-attention still maintained?
Actively maintained. The last commit to flash-attention was on Oct 20, 2026.
What can I use flash-attention for?
Training large Transformer language models with long sequences on NVIDIA A100 or H100 GPUs. Serving LLM inference with incremental decoding and a paged KV cache. Running attention on AMD Instinct GPUs through the ROCm backends.
How do I install flash-attention?
Install the build prerequisites: packaging, psutil and ninja Install FlashAttention from PyPI: pip install flash-attn --no-build-isolation
Where can I get flash-attention?
The source code and installation instructions are on GitHub: https://github.com/Dao-AILab/flash-attention

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects