Repolode
Submit
ModelTC

ModelTC/LightLLM

Python-based LLM inference and serving framework focused on a lightweight design, easy scaling and high throughput.

View on GitHub
Stars
4,358
Forks
383
Open issues
183
Last commit
yesterday

What is it?

LightLLM is an inference and serving framework for large language models, written in Python. It is designed to host LLMs behind a service endpoint while keeping the codebase small and easy to scale across hardware.

The project draws on several established open-source LLM serving implementations, including FasterTransformer, TGI, vLLM and FlashAttention. Its scheduler and KV cache management are described in published papers, covering topics such as past-future request scheduling under SLA targets and constrained decoding with pushdown automata for structured output.

The repository includes installation guides, a quick start and a DeepSeek deployment tutorial, along with a Docker image. Several projects, including vLLM and SGLang, reuse some of its kernels, and its pure-Python design is positioned as a base for research.

Key features

  • 01Lightweight, pure-Python codebase for LLM inference and serving.
  • 02Scheduler and KV cache management designed for high-throughput serving.
  • 03Support for constrained (structured) decoding.
  • 04Prefix KV cache transfer between data-parallel ranks.
  • 05Deployment tutorials, including DeepSeek models, and a Docker image.

Use cases

  • Serving open LLMs such as DeepSeek behind an inference endpoint.
  • Researching LLM scheduling or KV cache strategies on a readable Python base.
  • Running structured-output generation with constrained decoding.

Quick start

  1. 1

    Follow the installation guide linked in the README

  2. 2

    Follow the Quick Start guide in the English docs

In collections

Natural language processing tools & libraries

Topics

#llm#inference#model-serving#gpt#llama#deep-learning#openai-triton#nlp

FAQ

What is LightLLM?
Python-based LLM inference and serving framework focused on a lightweight design, easy scaling and high throughput. LightLLM is an inference and serving framework for large language models, written in Python. It is designed to host LLMs behind a service endpoint while keeping the codebase small and easy to scale across hardware.
Is LightLLM free to use? What license does it have?
LightLLM is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is LightLLM still maintained?
Actively maintained. The last commit to LightLLM was on Nov 6, 2026.
What can I use LightLLM for?
Serving open LLMs such as DeepSeek behind an inference endpoint. Researching LLM scheduling or KV cache strategies on a readable Python base. Running structured-output generation with constrained decoding.
How do I install LightLLM?
Follow the installation guide linked in the README Follow the Quick Start guide in the English docs
Where can I get LightLLM?
The source code and installation instructions are on GitHub: https://github.com/ModelTC/LightLLM

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects