triton-inference-server/tensorrtllm_backend

What is it?

Key features

• 01Serves TensorRT-LLM engines and HuggingFace models through Triton's HTTP and gRPC endpoints.
• 02Implements inflight batching and paged attention in its C++ backend.
• 03Configures KV cache, quantization and parallelism via a model.yaml file.
• 04Supports tensor, pipeline and expert parallelism for large models.
• 05Allows cancelling in-flight generation requests by request ID.
• 06Exposes TRT-LLM batch manager statistics on Triton's metrics endpoint.

Use cases

• Deploying a large LLM across multiple NVIDIA GPUs or nodes behind a Triton endpoint.
• Serving several models from one Triton server with a shared model repository.
• Benchmarking core TensorRT-LLM latency and throughput with the bundled scripts.
• Running Triton inference in Slurm-managed GPU clusters.

Quick start

1. 1

   Launch the NVIDIA Triton TensorRT-LLM container

   docker run --rm -it --net host --shm-size=2g --ulimit memlock=-1 --gpus all -v ~/.cache/huggingface:/root/.cache/huggingface nvcr.io/nvidia/tritonserver:25.12-trtllm-python-py3 bash
2. 2

   Clone the TensorRT-LLM repository and set your model in model.yaml

   git clone https://github.com/NVIDIA/TensorRT-LLM.git
3. 3

   Launch the Triton server from the parent directory of the clone

   python3 TensorRT-LLM/triton_backend/scripts/launch_triton_server.py --model_repo=TensorRT-LLM/triton_backend/all_models/llmapi/

Things to know

• !Requires NVIDIA GPUs and the TensorRT-LLM stack, so it does not run on ordinary hardware.
• !Setup relies on a large NVIDIA container image and Docker with GPU support.
• !Gated models such as Llama need a HuggingFace token.

In collections

Topics

FAQ

What is tensorrtllm_backend?

A Triton Inference Server backend that lets you serve TensorRT-LLM and HuggingFace-compatible LLMs over Triton's HTTP and gRPC APIs. TensorRT-LLM Backend connects NVIDIA's TensorRT-LLM inference library to Triton Inference Server, so LLMs can be deployed and queried through Triton's standard model repository and inference endpoints. It includes an inflight batching C++ implementation that supports paged attention, along with a PyTorch-based path built on the LLM API that can serve HuggingFace models without compiling an engine first.

Is tensorrtllm_backend free to use? What license does it have?

tensorrtllm_backend is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.

Is tensorrtllm_backend still maintained?

Actively maintained. The last commit to tensorrtllm_backend was on Nov 2, 2026.

What can I use tensorrtllm_backend for?

Deploying a large LLM across multiple NVIDIA GPUs or nodes behind a Triton endpoint. Serving several models from one Triton server with a shared model repository. Benchmarking core TensorRT-LLM latency and throughput with the bundled scripts. Running Triton inference in Slurm-managed GPU clusters.

How do I install tensorrtllm_backend?

Launch the NVIDIA Triton TensorRT-LLM container: docker run --rm -it --net host --shm-size=2g --ulimit memlock=-1 --gpus all -v ~/.cache/huggingface:/root/.cache/huggingface nvcr.io/nvidia/tritonserver:25.12-trtllm-python-py3 bash Clone the TensorRT-LLM repository and set your model in model.yaml: git clone https://github.com/NVIDIA/TensorRT-LLM.git Launch the Triton server from the parent directory of the clone: python3 TensorRT-LLM/triton_backend/scripts/launch_triton_server.py --model_repo=TensorRT-LLM/triton_backend/all_models/llmapi/

Where can I get tensorrtllm_backend?

The source code and installation instructions are on GitHub: https://github.com/triton-inference-server/tensorrtllm_backend