Repolode
Submit
InternLM

InternLM/lmdeploy

A toolkit for compressing, deploying, and serving LLMs and vision-language models, with TurboMind and PyTorch inference engines.

View on GitHubWebsite
Stars
8,111
Forks
767
Open issues
613
Last commit
4 hours ago

What is it?

LMDeploy is a toolkit for compressing, deploying, and serving large language models. It provides an offline inference pipeline for Python scripts and an API server for online serving, and supports multi-model and multi-machine, multi-GPU deployments.

The project includes two inference engines. TurboMind focuses on performance and adds features such as persistent batching, blocked KV cache, tensor parallelism, and weight-only and KV cache quantization (including AWQ 4-bit). The PyTorch engine is written entirely in Python, which makes it easier to modify and experiment with new models. A support matrix lists which models each engine handles.

The supported model list covers many families, including Llama, Qwen, InternLM, DeepSeek, Mixtral, and GLM for text, plus vision-language models such as LLaVA, Qwen2-VL, and InternVL. The toolkit also supports Huawei Ascend hardware and NVIDIA GPUs.

Key features

  • 01Offline batch inference through a Python pipeline API.
  • 02Online serving with an OpenAI-compatible API server.
  • 03Weight-only and KV cache quantization, including 4-bit AWQ.
  • 04Two inference engines: TurboMind for performance and PyTorch for Python-based development.
  • 05Multi-model, multi-machine, multi-GPU deployment with a request proxy server.

Use cases

  • Serving an open-source LLM or VLM behind an internal API to reduce GPU memory and latency.
  • Running quantized models on limited NVIDIA GPUs.
  • Deploying models on Huawei Ascend hardware.
  • Prototyping new model architectures with the Python-based PyTorch engine.

Quick start

  1. 1

    Create a conda environment with Python 3.12

    conda create -n lmdeploy python=3.12 -y
  2. 2

    Activate the conda environment

    conda activate lmdeploy
  3. 3

    Install LMDeploy from PyPI

    pip install lmdeploy
  4. 4

    Run offline batch inference with a pipeline

Things to know

  • Prebuilt PyPI wheels target CUDA 12.8, so GPU setups may need matching drivers.
  • Many features and models are tied to NVIDIA GPUs, and support varies by engine and platform.

Topics

#llm-inference#llm#quantization#model-serving#gpu#cuda#openai-api#vision-language-model#codellama#cuda-kernels

FAQ

What is lmdeploy?
A toolkit for compressing, deploying, and serving LLMs and vision-language models, with TurboMind and PyTorch inference engines. LMDeploy is a toolkit for compressing, deploying, and serving large language models. It provides an offline inference pipeline for Python scripts and an API server for online serving, and supports multi-model and multi-machine, multi-GPU deployments.
Is lmdeploy free to use? What license does it have?
lmdeploy is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is lmdeploy still maintained?
Actively maintained. The last commit to lmdeploy was on Oct 21, 2026.
What can I use lmdeploy for?
Serving an open-source LLM or VLM behind an internal API to reduce GPU memory and latency. Running quantized models on limited NVIDIA GPUs. Deploying models on Huawei Ascend hardware. Prototyping new model architectures with the Python-based PyTorch engine.
How do I install lmdeploy?
Create a conda environment with Python 3.12: conda create -n lmdeploy python=3.12 -y Activate the conda environment: conda activate lmdeploy Install LMDeploy from PyPI: pip install lmdeploy Run offline batch inference with a pipeline
Where can I get lmdeploy?
The source code and installation instructions are on GitHub: https://github.com/InternLM/lmdeploy

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects