Repolode
Submit
facebookincubator

facebookincubator/AITemplate

A Python framework that compiles deep neural networks into fp16 CUDA/HIP C++ code for fast inference on NVIDIA and AMD GPUs.

Open-source alternative to:TensorRT

View on GitHub
Stars
4,735
Forks
386
Open issues
163
Last commit
last month

What is it?

AITemplate (AIT) is a Python framework that turns deep neural networks into CUDA (NVIDIA) or HIP (AMD) C++ code for inference serving. It targets fp16 workloads on TensorCore and MatrixCore hardware, and the project lists ResNet, MaskRCNN, BERT, Vision Transformer and Stable Diffusion among the models it covers.

The compiler applies horizontal fusion (combining parallel GEMM and LayerNorm operators into one kernel), vertical fusion (folding elementwise operations, reductions and layout changes into TensorCore/MatrixCore work) and memory fusion (merging concatenation, split and slice with preceding operators). Each compiled model is a self-contained binary that does not depend on cuBLAS, cuDNN, TensorRT or similar runtimes.

The Python runtime accepts PyTorch tensors without extra copies, and a companion tool, FX2AIT, converts PyTorch models and can partially accelerate models that contain unsupported operators. The project requires SM80+ NVIDIA GPUs or CDNA2 AMD GPUs for tested configurations.

Key features

  • 01Compiles fp16 neural networks into CUDA/HIP C++ targeting TensorCore and MatrixCore hardware.
  • 02Applies horizontal, vertical and memory operator fusion to reduce kernel launches.
  • 03Produces self-contained model binaries without cuBLAS, cuDNN or TensorRT dependencies.
  • 04Python runtime accepts PyTorch tensors directly without extra copies.
  • 05FX2AIT converts PyTorch models and partially accelerates unsupported operators.

Use cases

  • Serving ResNet, BERT or Vision Transformer models in fp16 on NVIDIA A100-class GPUs.
  • Running Stable Diffusion inference on AMD MI-series GPUs.
  • Accelerating PyTorch models that contain a few operators AIT does not support.

Quick start

  1. 1

    Clone the repository with its submodules

    git clone --recursive https://github.com/facebookincubator/AITemplate
  2. 2

    Change into the python directory

    cd python
  3. 3

    Build the Python wheel

    python setup.py bdist_wheel
  4. 4

    Install the built wheel

    pip install dist/*.whl --force-reinstall

Things to know

  • NVIDIA support is only tested on SM80+ GPUs such as Ampere, and older T4 or V100 GPUs may not work with all kernels.
  • AMD support is only tested on CDNA2 GPUs (MI-210/250), and older CDNA1 GPUs may have compiler issues.
  • Building from source requires the correct CUDA or ROCm compiler, since an incorrect one causes performance regressions.

In collections

Run and train AI on GPUsCompilers, transpilers & programming languages

Topics

#inference#gpu#cuda#fp16#compiler#pytorch#deep-learning

FAQ

What is AITemplate?
A Python framework that compiles deep neural networks into fp16 CUDA/HIP C++ code for fast inference on NVIDIA and AMD GPUs. AITemplate (AIT) is a Python framework that turns deep neural networks into CUDA (NVIDIA) or HIP (AMD) C++ code for inference serving. It targets fp16 workloads on TensorCore and MatrixCore hardware, and the project lists ResNet, MaskRCNN, BERT, Vision Transformer and Stable Diffusion among the models it covers.
Is AITemplate free to use? What license does it have?
AITemplate is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is AITemplate still maintained?
Updated within 6 months. The last commit to AITemplate was on Oct 7, 2026.
What can I use AITemplate for?
Serving ResNet, BERT or Vision Transformer models in fp16 on NVIDIA A100-class GPUs. Running Stable Diffusion inference on AMD MI-series GPUs. Accelerating PyTorch models that contain a few operators AIT does not support.
How do I install AITemplate?
Clone the repository with its submodules: git clone --recursive https://github.com/facebookincubator/AITemplate Change into the python directory: cd python Build the Python wheel: python setup.py bdist_wheel Install the built wheel: pip install dist/*.whl --force-reinstall
Where can I get AITemplate?
The source code and installation instructions are on GitHub: https://github.com/facebookincubator/AITemplate

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects