triton-inference-server/server

What is it?

Key features

• 01Serves models from TensorRT, PyTorch, ONNX, OpenVINO, Python and other frameworks.
• 02Dynamic batching and sequence batching for stateful models.
• 03Concurrent model execution across GPUs and CPUs.
• 04HTTP/REST and gRPC endpoints following the KServe v2 protocol.
• 05Model ensembles and Business Logic Scripting for multi-model pipelines.
• 06Custom backends in C/C++ or Python, plus C and Java APIs for in-process use.
• 07Metrics for GPU utilization, throughput and latency.

Use cases

• Serving production inference endpoints for models trained in several frameworks.
• Running multi-model pipelines that chain preprocessing, inference and postprocessing.
• Deploying models on Kubernetes or edge devices with GPU or CPU hardware.
• Embedding model inference directly into a C++ or Java application.

Quick start

1. 1

   Clone the repository and fetch example models

   git clone -b r26.09 https://github.com/triton-inference-server/server.git
2. 2

   Launch Triton from the NGC Triton container

   docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:26.09-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx
3. 3

   Send an inference request with the image_client example from the NGC Triton SDK container

   docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:26.09-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg

Things to know

• !Running the quick start requires an NVIDIA GPU for the main example, with a CPU-only option documented separately.
• !The README describes the main branch as under development; the stable release is version 2.73.0.
• !Recommended deployment is through NVIDIA's Docker containers, so Docker is needed for the easiest setup.

Topics

FAQ

What is server?

Open-source inference server that deploys AI models from TensorRT, PyTorch, ONNX, OpenVINO and other frameworks over HTTP/REST and gRPC. Triton Inference Server is an open source serving system for AI models. It lets teams deploy models from multiple frameworks, including TensorRT, PyTorch, ONNX, OpenVINO and Python, behind a single server. Supported targets include NVIDIA GPUs, x86 and ARM CPUs, and AWS Inferentia, spanning cloud, data center, edge and embedded devices.

Is server free to use? What license does it have?

server is open source under the BSD-3-Clause license. Check the license terms before using it in commercial products.

Is server still maintained?

Actively maintained. The last commit to server was on Oct 22, 2026.

What can I use server for?

Serving production inference endpoints for models trained in several frameworks. Running multi-model pipelines that chain preprocessing, inference and postprocessing. Deploying models on Kubernetes or edge devices with GPU or CPU hardware. Embedding model inference directly into a C++ or Java application.

How do I install server?

Clone the repository and fetch example models: git clone -b r26.09 https://github.com/triton-inference-server/server.git Launch Triton from the NGC Triton container: docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:26.09-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx Send an inference request with the image_client example from the NGC Triton SDK container: docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:26.09-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg

Where can I get server?

The source code and installation instructions are on GitHub: https://github.com/triton-inference-server/server