Repolode
Submit
alibaba

alibaba/rtp-llm

A high-performance LLM inference engine from Alibaba, used across its own business units for serving large language models.

View on GitHub
Stars
1,394
Forks
299
Open issues
274
Last commit
10 hours ago

What is it?

RTP-LLM is an inference acceleration engine for large language models, developed by Alibaba's Foundation Model Inference Team. It is designed to serve LLMs efficiently in production and is used across several Alibaba business units, including Taobao, Tmall, and Amap.

The engine uses optimized CUDA kernels such as PagedAttention and FlashAttention, supports weight-only INT8 and INT4 quantization (including GPTQ and AWQ checkpoints), and adaptive KV cache quantization. It loads HuggingFace-format models in SafeTensors, PyTorch, and Megatron weight formats, and can run multi-machine, multi-GPU tensor parallelism.

Serving features include serving multiple LoRA adapters from a single model instance, multimodal inputs combining images and text, prefix and system prompt caching for multi-turn dialogues, speculative decoding, and Prefill/Decode separation. The project is a sub-project of havenask and is built on ideas from FasterTransformer and TensorRT-LLM kernels.

Key features

  • 01Optimized CUDA kernels including PagedAttention and FlashAttention
  • 02WeightOnly INT8/INT4 quantization, compatible with GPTQ and AWQ
  • 03Multi-machine, multi-GPU tensor parallelism
  • 04Serves multiple LoRA adapters from one model instance
  • 05Context prefix cache and system prompt cache for multi-turn dialogues
  • 06Speculative decoding and Prefill/Decode separation

Use cases

  • Deploying high-throughput LLM inference services on NVIDIA GPUs
  • Serving many LoRA fine-tunes of one base model
  • Running multimodal image-and-text models in production
  • Reducing GPU memory and latency with quantization and KV cache optimizations

Quick start

  1. 1

    Follow the install guide on the RTP-LLM documentation site

  2. 2

    Follow the Quick Start guide to send your first request

Things to know

  • The README names CUDA kernels and V100 optimization, so NVIDIA GPU setups are the main focus.
  • The README says AMD ROCm and Intel CPU support are still in development.

In collections

Run LLMs locallyRun and train AI on GPUs

Topics

#llm-inference#llm-serving#gpu#cuda#quantization#model-serving#lora#gpt#inference#llama

FAQ

What is rtp-llm?
A high-performance LLM inference engine from Alibaba, used across its own business units for serving large language models. RTP-LLM is an inference acceleration engine for large language models, developed by Alibaba's Foundation Model Inference Team. It is designed to serve LLMs efficiently in production and is used across several Alibaba business units, including Taobao, Tmall, and Amap.
Is rtp-llm free to use? What license does it have?
rtp-llm is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is rtp-llm still maintained?
Actively maintained. The last commit to rtp-llm was on Nov 6, 2026.
What can I use rtp-llm for?
Deploying high-throughput LLM inference services on NVIDIA GPUs Serving many LoRA fine-tunes of one base model Running multimodal image-and-text models in production Reducing GPU memory and latency with quantization and KV cache optimizations
How do I install rtp-llm?
Follow the install guide on the RTP-LLM documentation site Follow the Quick Start guide to send your first request
Where can I get rtp-llm?
The source code and installation instructions are on GitHub: https://github.com/alibaba/rtp-llm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects