Repolode
Submit
Tencent

Tencent/KsanaLLM

A high-performance, easy-to-use engine for LLM inference and serving, with an OpenAI-compatible API server.

View on GitHub
Stars
552
Forks
46
Open issues
14
Last commit
2 months ago

What is it?

KsanaLLM is an engine for running large language model inference and serving them over an API. It loads popular Hugging Face models such as LLaMA, Qwen, Baichuan, Yi and DeepSeek, and supports PyTorch and SafeTensors weight formats.

The engine focuses on throughput. It uses PagedAttention for attention key-value memory, dynamic batching with scheduling and memory tuning, prefix caching, and multi-GPU tensor parallelism. It also supports streaming output and decoding modes such as parallel sampling and beam search, and exposes a core /v1/chat/completions endpoint with tool-call and reasoning parsers.

KsanaLLM targets NVIDIA GPUs and Huawei Ascend NPUs, and the README also lists Kunlunxin XPU support. The project includes Docker build files, multi-node distributed serving, PD (prefill/decode) disaggregation mode, and an optional global KV cache connector backed by Mooncake Store.

Key features

  • 01Serves an OpenAI-compatible /v1/chat/completions API with streaming output.
  • 02Uses PagedAttention and prefix caching to manage attention KV memory.
  • 03Performs dynamic batching with tuned task scheduling and memory use.
  • 04Supports multi-GPU tensor parallelism and multi-node distributed execution.
  • 05Loads Hugging Face models in PyTorch and SafeTensors formats.
  • 06Runs on NVIDIA GPUs and Huawei Ascend NPUs, with Kunlunxin XPU listed.

Use cases

  • Serving open-weight LLMs such as LLaMA or Qwen behind an OpenAI-style API.
  • Running DeepSeek-class models across multiple GPUs or nodes.
  • Deploying LLM inference on Huawei Ascend NPU clusters.
  • Experimenting with prefill/decode disaggregation or global KV cache sharing.

In collections

Run LLMs locally

Topics

#llm-inference#llm-serving#cuda#pagedattention#tensor-parallelism#openai-api#ascend-npu#huggingface

FAQ

What is KsanaLLM?
A high-performance, easy-to-use engine for LLM inference and serving, with an OpenAI-compatible API server. KsanaLLM is an engine for running large language model inference and serving them over an API. It loads popular Hugging Face models such as LLaMA, Qwen, Baichuan, Yi and DeepSeek, and supports PyTorch and SafeTensors weight formats.
Is KsanaLLM free to use? What license does it have?
KsanaLLM is open source under the Other license. Check the license terms before using it in commercial products.
Is KsanaLLM still maintained?
Updated within 6 months. The last commit to KsanaLLM was on Aug 27, 2026.
What can I use KsanaLLM for?
Serving open-weight LLMs such as LLaMA or Qwen behind an OpenAI-style API. Running DeepSeek-class models across multiple GPUs or nodes. Deploying LLM inference on Huawei Ascend NPU clusters. Experimenting with prefill/decode disaggregation or global KV cache sharing.
Where can I get KsanaLLM?
The source code and installation instructions are on GitHub: https://github.com/Tencent/KsanaLLM

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects