Repolode
Submit
sgl-project

sgl-project/sglang

SGLang is an open-source inference framework for serving large language, vision-language, and diffusion models on many GPU, TPU, and CPU platforms.

View on GitHubWebsite
Stars
37,472
Forks
9,720
Open issues
5,965
Last commit
6 hours ago

What is it?

SGLang is a serving framework for large language models, vision-language models, and diffusion models. It is aimed at agentic workloads, large-scale online serving, and rollout generation for reinforcement learning. Users can start a model with a Docker image or a Python package installed via uv.

The project covers a broad hardware range, including NVIDIA GPUs from RTX consumer cards to B200-class data center chips, AMD Instinct GPUs, Google TPUs, Intel Arc GPUs and Xeon CPUs, Apple Silicon through Metal and MLX, and Huawei Ascend NPUs. Built-in SGLang Diffusion adds image and video generation in the same package.

Around the core engine, the project maintains related repositories for speculative decoding training, audio serving, TPU backends, and education, and it integrates with RL training frameworks such as verl and slime as well as orchestration tools such as Ray Serve and NVIDIA Dynamo.

Key features

  • 01Serves large language, vision-language, and diffusion models from one framework.
  • 02Supports NVIDIA, AMD, Google TPU, Intel, Apple Silicon, Huawei Ascend, and Moore Threads hardware.
  • 03Includes SGLang Diffusion for image and video generation.
  • 04Provides a Docker image and a pip/uv-installable Python package.
  • 05Integrates with RL training frameworks for rollout generation.

Use cases

  • Deploying an LLM or VLM as an online inference service on GPU clusters.
  • Generating rollouts for reinforcement learning post-training.
  • Running model inference on Apple Silicon Macs or Intel hardware.
  • Serving image or video generation models with diffusion support.

Quick start

  1. 1

    Pull the Docker image with SGLang and its dependencies

    docker pull lmsysorg/sglang:latest
  2. 2

    Or install SGLang with uv in an activated Python environment

    uv pip install --prerelease=allow sglang
  3. 3

    Launch your model using the Quickstart guide in the docs

Things to know

  • The uv install requires allowing pre-release packages via the --prerelease flag.
  • Several hardware integrations, such as AWS Trainium and Cambricon MLU, are still in progress.

Topics

#llm#inference#gpu#serving#vlm#diffusion#cuda#moe#attention#blackwell

FAQ

What is sglang?
SGLang is an open-source inference framework for serving large language, vision-language, and diffusion models on many GPU, TPU, and CPU platforms. SGLang is a serving framework for large language models, vision-language models, and diffusion models. It is aimed at agentic workloads, large-scale online serving, and rollout generation for reinforcement learning. Users can start a model with a Docker image or a Python package installed via uv.
Is sglang free to use? What license does it have?
sglang is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is sglang still maintained?
Actively maintained. The last commit to sglang was on Oct 20, 2026.
What can I use sglang for?
Deploying an LLM or VLM as an online inference service on GPU clusters. Generating rollouts for reinforcement learning post-training. Running model inference on Apple Silicon Macs or Intel hardware. Serving image or video generation models with diffusion support.
How do I install sglang?
Pull the Docker image with SGLang and its dependencies: docker pull lmsysorg/sglang:latest Or install SGLang with uv in an activated Python environment: uv pip install --prerelease=allow sglang Launch your model using the Quickstart guide in the docs
Where can I get sglang?
The source code and installation instructions are on GitHub: https://github.com/sgl-project/sglang

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects