xlite-dev/LeetCUDA

What is it?

Key features

• 01Over 200 CUDA kernels, from element-wise operations to Hard++ level attention
• 02HGEMM implementations using WMMA, MMA PTX and CuTe with cuBLAS comparison benchmarks
• 03FlashAttention-2 variants built on Tensor Cores with pure MMA PTX
• 04Kernels covering BF16, FP8 and FP4 data types
• 05Triton and CUTLASS/CuTe examples alongside raw CUDA code
• 06Each kernel paired with PyTorch Python bindings and tests

Use cases

• Learning CUDA kernel programming step by step from basic to advanced topics
• Studying Tensor Core matrix multiplication and FlashAttention implementations
• Benchmarking custom GEMM or attention kernels against cuBLAS or SDPA
• Giving a coding agent CUDA reference examples through the bundled skill

Quick start

1. 1

   Clone the repository and initialize submodules

   git clone https://github.com/xlite-dev/LeetCUDA.git && cd LeetCUDA
2. 2

   Update submodules and enter the interview kernels directory

   git submodule update --init --recursive --force && cd kernels/interview
3. 3

   Install the latest cuBLAS and cuDNN for benchmarks (replace the old versions as shown in the README)

   apt install -y cublas-cuda-13 cudnn9-cuda-13 && apt install -y ccache
4. 4

   Build for Blackwell GPUs and run the benchmark

   ./build.sh --arch sm_120a && ./bin/leetcuda_bench_sm120a.bin --bench

Things to know

• !Requires NVIDIA GPU with CUDA toolkit and recent cuBLAS/cuDNN for the benchmarks.
• !The quick start build targets Blackwell (sm_120a) by example; other architectures need different build flags.

Topics

FAQ

What is LeetCUDA?

An open-source book and code collection of modern CUDA learning notes, with 200+ kernels covering HGEMM, FP8/FP4 GEMM, and Flash-Attention. LeetCUDA is a learning resource for writing GPU kernels in CUDA, organized as an open-source book with an accompanying code repository. It walks readers from simple element-wise and reduction kernels through matrix multiplication and attention, with each topic following the same workflow: a CUDA kernel implementation, PyTorch Python bindings, and tests.

Is LeetCUDA free to use? What license does it have?

LeetCUDA is open source under the GPL-3.0 license. Check the license terms before using it in commercial products.

Is LeetCUDA still maintained?

Actively maintained. The last commit to LeetCUDA was on Oct 20, 2026.

What can I use LeetCUDA for?

Learning CUDA kernel programming step by step from basic to advanced topics Studying Tensor Core matrix multiplication and FlashAttention implementations Benchmarking custom GEMM or attention kernels against cuBLAS or SDPA Giving a coding agent CUDA reference examples through the bundled skill

How do I install LeetCUDA?

Clone the repository and initialize submodules: git clone https://github.com/xlite-dev/LeetCUDA.git && cd LeetCUDA Update submodules and enter the interview kernels directory: git submodule update --init --recursive --force && cd kernels/interview Install the latest cuBLAS and cuDNN for benchmarks (replace the old versions as shown in the README): apt install -y cublas-cuda-13 cudnn9-cuda-13 && apt install -y ccache Build for Blackwell GPUs and run the benchmark: ./build.sh --arch sm_120a && ./bin/leetcuda_bench_sm120a.bin --bench

Where can I get LeetCUDA?

The source code and installation instructions are on GitHub: https://github.com/xlite-dev/LeetCUDA