Repolode
Submit
fla-org

fla-org/flash-linear-attention

Triton-based kernels and training-ready layers for linear attention, sparse attention, state space and hybrid LLM architectures.

View on GitHubWebsite
Stars
5,952
Forks
778
Open issues
126
Last commit
4 hours ago

What is it?

Flash Linear Attention (fla) is a Python library of hardware-efficient building blocks for sequence models that replace or complement standard softmax attention. It provides Triton kernels, ready-made layers such as GatedDeltaNet, RetNet, GLA and Mamba2, and 🤗 Transformers-compatible model classes built on them. The project states that its implementations are platform-agnostic and verified on NVIDIA, AMD and Intel hardware.

Beyond individual layers, fla includes fused modules such as RMSNorm with gating, fused linear cross entropy and rotary embeddings, aimed at reducing memory use and speeding up training. Hybrid models can mix standard attention layers into linear attention models through a configuration argument, with per-layer control. Generation works through the standard Hugging Face APIs, and pretrained checkpoints are hosted under the fla-hub organization.

The repository also documents a training path through the companion flame framework built on torchtitan, plus evaluation scripts that use lm-evaluation-harness and perplexity measurement.

Key features

  • 01Triton kernels for linear attention, sparse attention and state space model variants.
  • 02Drop-in token mixing layers such as MultiScaleRetention that replace multihead attention.
  • 03Hugging Face Transformers-compatible model classes with configurable configs.
  • 04Hybrid models that insert standard attention layers at chosen depths.
  • 05Fused norm, rotary embedding and cross entropy modules to reduce memory and speed up training.
  • 06Backend extras for CUDA, ROCm, XPU, Ascend NPU and CPU.

Use cases

  • Replacing quadratic attention in a long-context language model with a linear attention variant.
  • Pretraining or fine-tuning subquadratic LLMs with a Hugging Face-style workflow.
  • Experimenting with hybrid architectures that mix attention and recurrent layers.
  • Benchmarking kernel speed for linear attention on different GPUs.

Topics

#linear-attention#triton#large-language-models#sequence-modeling#state-space-models#pytorch#gpu-kernels#machine-learning-systems#natural-language-processing

FAQ

What is flash-linear-attention?
Triton-based kernels and training-ready layers for linear attention, sparse attention, state space and hybrid LLM architectures. Flash Linear Attention (fla) is a Python library of hardware-efficient building blocks for sequence models that replace or complement standard softmax attention. It provides Triton kernels, ready-made layers such as GatedDeltaNet, RetNet, GLA and Mamba2, and 🤗 Transformers-compatible model classes built on them. The project states that its implementations are platform-agnostic and verified on NVIDIA, AMD and Intel hardware.
Is flash-linear-attention free to use? What license does it have?
flash-linear-attention is open source under the MIT license. Check the license terms before using it in commercial products.
Is flash-linear-attention still maintained?
Actively maintained. The last commit to flash-linear-attention was on Oct 20, 2026.
What can I use flash-linear-attention for?
Replacing quadratic attention in a long-context language model with a linear attention variant. Pretraining or fine-tuning subquadratic LLMs with a Hugging Face-style workflow. Experimenting with hybrid architectures that mix attention and recurrent layers. Benchmarking kernel speed for linear attention on different GPUs.
Where can I get flash-linear-attention?
The source code and installation instructions are on GitHub: https://github.com/fla-org/flash-linear-attention

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects