Efficient Triton GPU kernels for LLM training that fuse common layers and losses to cut memory use and raise throughput.
6.7K627PythonBSD-2-ClauseUpdated 47 minutes ago
flagos-ai/FlagGems
An operator library of PyTorch-compatible LLM kernels written in Triton, designed to run across multiple AI accelerator backends.
- Stars
- 1,160
- Forks
- 588
- Open issues
- 1,113
- Last commit
- 59 minutes ago
What is it?
FlagGems is an operator library for large language models, implemented in the Triton language. It provides a collection of backend-neutral kernels intended to speed up LLM training and inference across different hardware platforms. It is part of the FlagOS system software stack.
FlagGems registers with the ATen backend of PyTorch, so existing PyTorch code can keep using familiar APIs while the underlying operators run on Triton kernels. The project offers a large set of PyTorch-compatible operators, hand-optimized kernels for selected operators, and automatic code generation for pointwise operators. It works in eager mode and does not depend on torch.compile.
The library includes a multi-backend interface for diverse hardware and lists support for more than ten backends. The README names Bert-base-uncased, Llama-2-7b and Llava-1.5-7b as sample models used for testing.
Key features
- 01Large collection of PyTorch-compatible operators
- 02Hand-optimized kernels for selected operators
- 03Eager-mode support, independent of torch.compile
- 04Automatic code generation for pointwise operators
- 05Multi-backend interface covering more than ten hardware backends
Use cases
- Running LLM training or inference on non-NVIDIA AI accelerators without rewriting PyTorch code
- Porting PyTorch workloads across different chip software stacks
- Writing or benchmarking Triton kernels against a PyTorch operator set
Topics
FAQ
- What is FlagGems?
- An operator library of PyTorch-compatible LLM kernels written in Triton, designed to run across multiple AI accelerator backends. FlagGems is an operator library for large language models, implemented in the Triton language. It provides a collection of backend-neutral kernels intended to speed up LLM training and inference across different hardware platforms. It is part of the FlagOS system software stack.
- Is FlagGems free to use? What license does it have?
- FlagGems is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
- Is FlagGems still maintained?
- Actively maintained. The last commit to FlagGems was on Oct 22, 2026.
- What can I use FlagGems for?
- Running LLM training or inference on non-NVIDIA AI accelerators without rewriting PyTorch code Porting PyTorch workloads across different chip software stacks Writing or benchmarking Triton kernels against a PyTorch operator set
- Where can I get FlagGems?
- The source code and installation instructions are on GitHub: https://github.com/flagos-ai/FlagGems
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
Triton-based kernels and training-ready layers for linear attention, sparse attention, state space and hybrid LLM architectures.
AI & Machine Learning#linear-attention#triton#large-language-models#sequence-modeling#state-space-models
6K779PythonMITUpdated 11 hours ago
A cross-platform accelerator for running ML models exported to ONNX format, with training support for PyTorch scripts.
22.1K4.4KC++MITUpdated 48 minutes ago
Open-weight gpt-oss-120b and gpt-oss-20b language models from OpenAI, with reference inference, tool, and client code.
20.5K2.2KPythonApache-2.0Updated 10 days ago
A personal study and engineering notebook of hand-written CUDA kernels, CUTLASS/CuTe, Triton examples and LLM optimization notes.
3.3K297CudaNo licenseUpdated 11 days ago
Model-definition framework for state-of-the-art text, vision, audio and multimodal machine learning models, for inference and training.
168K34.9KPythonApache-2.0Updated 3 hours ago