Repolode
Submit
Lightning-AI

Lightning-AI/lightning-thunder

A source-to-source compiler for PyTorch that optimizes models for training and inference with inspectable IR and composable transforms.

View on GitHub
Stars
1,474
Forks
120
Open issues
483
Last commit
26 days ago

What is it?

Thunder is a deep learning compiler for PyTorch that takes a model and transforms it into an optimized version. It captures the full computation as a simple Python-like intermediate representation, which can be inspected, profiled, and rewritten to swap in faster operations or kernels.

Optimizations are applied through transforms and plugins that work on the computation trace, the model, and its weights. Supported options include distributed parallelism (DDP, FSDP, TP), FP8 and MXFP8 precision, quantization, and CUDA Graphs. Execution is routed to fusers such as NVFuser and torch.compile, specialized libraries like cuDNN SDPA and TransformerEngine, custom Triton and CUDA kernels, or PyTorch eager operations.

The entry point is thunder.compile(model), which can be applied to a plain torch module or to models from libraries such as LitGPT and Hugging Face Transformers.

Key features

  • 01Captures full computation graphs as an inspectable Python-like IR.
  • 02Composable transforms that modify the model, trace, and weights together.
  • 03Plugins for distributed training (DDP, FSDP, TP), FP8/MXFP8, quantization, and CUDA Graphs.
  • 04Dispatches ops to fusers, cuDNN, TransformerEngine, Triton, or CUDA kernels.

Use cases

  • Speeding up LLM pretraining or fine-tuning on NVIDIA GPUs.
  • Running Hugging Face or torchvision models with lower-latency inference.
  • Experimenting with mixed precision or quantization settings on a model.
  • Scaling a single-GPU PyTorch model to multi-GPU distributed execution.

Quick start

  1. 1

    Install Thunder via pip

    pip install lightning-thunder
  2. 2

    Install or upgrade PyTorch and torchvision

    pip install -U torch torchvision
  3. 3

    Compile a PyTorch model with Thunder and run it (see the hello world example in the README)

Things to know

  • Optional NVIDIA GPU support requires extra packages such as nvfuser builds matched to specific torch and CUDA versions.
  • Float8 support via transformer_engine must be compiled from source and is slow to install.
  • The README notes Thunder is not a replacement for torch.compile and gains speed mainly in specific setups.

Topics

#pytorch#deep-learning-compiler#llm#training#inference#cuda#distributed-training#quantization

FAQ

What is lightning-thunder?
A source-to-source compiler for PyTorch that optimizes models for training and inference with inspectable IR and composable transforms. Thunder is a deep learning compiler for PyTorch that takes a model and transforms it into an optimized version. It captures the full computation as a simple Python-like intermediate representation, which can be inspected, profiled, and rewritten to swap in faster operations or kernels.
Is lightning-thunder free to use? What license does it have?
lightning-thunder is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is lightning-thunder still maintained?
Actively maintained. The last commit to lightning-thunder was on Oct 12, 2026.
What can I use lightning-thunder for?
Speeding up LLM pretraining or fine-tuning on NVIDIA GPUs. Running Hugging Face or torchvision models with lower-latency inference. Experimenting with mixed precision or quantization settings on a model. Scaling a single-GPU PyTorch model to multi-GPU distributed execution.
How do I install lightning-thunder?
Install Thunder via pip: pip install lightning-thunder Install or upgrade PyTorch and torchvision: pip install -U torch torchvision Compile a PyTorch model with Thunder and run it (see the hello world example in the README)
Where can I get lightning-thunder?
The source code and installation instructions are on GitHub: https://github.com/Lightning-AI/lightning-thunder

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects