Repolode
Submit
NVIDIA

NVIDIA/TensorRT-LLM

An open-source library from NVIDIA for optimizing LLM and visual generation inference on NVIDIA GPUs, with a Python API and C++/Python runtimes.

View on GitHubWebsite
Stars
14,937
Forks
2,862
Open issues
1,789
Last commit
11 hours ago

What is it?

TensorRT LLM is a library for running large language model inference efficiently on NVIDIA GPUs. It provides a Python API for defining LLMs, custom kernels for common operations such as attention, GEMMs and MoE layers, and runtime components that orchestrate execution in both Python and C++.

The project covers single-GPU, multi-GPU and multi-node setups, with support for parallelism strategies, prefill-decode disaggregation, wide expert parallelism and speculative decoding. It integrates with NVIDIA Dynamo and the Triton Inference Server, and its PyTorch-native architecture lets developers modify the runtime or adapt pre-defined models using PyTorch code.

The repository also includes visual generation (diffusion) model support. Telemetry is collected by default but can be disabled through an environment variable, a file, the Python API, or a CLI flag.

Key features

  • 01Custom kernels for attention, GEMM and MoE operations on NVIDIA GPUs
  • 02Python LLM API spanning single-GPU to multi-node deployments
  • 03Parallelism options including expert parallelism and prefill-decode disaggregation
  • 04Speculative decoding and other runtime optimizations
  • 05PyTorch-native architecture for modifying models and runtime
  • 06Integration with NVIDIA Dynamo and Triton Inference Server

Use cases

  • Serving large language models with low latency and high throughput on NVIDIA data center GPUs
  • Deploying quantized LLMs such as FP8 or FP4 models on Blackwell hardware
  • Running multi-GPU or multi-node inference for mixture-of-experts models
  • Accelerating visual generation models on NVIDIA GPUs

Things to know

  • Requires NVIDIA GPUs, so it cannot run on other hardware.
  • The license metadata reads NOASSERTION, though the README badge says Apache 2, so verify the license before use.
  • The current release is a release candidate (1.4.0rc2), so APIs may change.

In collections

Run LLMs locallyRun and train AI on GPUs

Topics

#llm-inference#gpu#cuda#llm-serving#moe#pytorch#inference-optimization#blackwell

FAQ

What is TensorRT-LLM?
An open-source library from NVIDIA for optimizing LLM and visual generation inference on NVIDIA GPUs, with a Python API and C++/Python runtimes. TensorRT LLM is a library for running large language model inference efficiently on NVIDIA GPUs. It provides a Python API for defining LLMs, custom kernels for common operations such as attention, GEMMs and MoE layers, and runtime components that orchestrate execution in both Python and C++.
Is TensorRT-LLM free to use? What license does it have?
TensorRT-LLM is open source under the Other license. Check the license terms before using it in commercial products.
Is TensorRT-LLM still maintained?
Actively maintained. The last commit to TensorRT-LLM was on Oct 24, 2026.
What can I use TensorRT-LLM for?
Serving large language models with low latency and high throughput on NVIDIA data center GPUs Deploying quantized LLMs such as FP8 or FP4 models on Blackwell hardware Running multi-GPU or multi-node inference for mixture-of-experts models Accelerating visual generation models on NVIDIA GPUs
Where can I get TensorRT-LLM?
The source code and installation instructions are on GitHub: https://github.com/NVIDIA/TensorRT-LLM

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects