microsoft/vidur

What is it?

Key features

• 01Estimates TTFT, TPOT, end-to-end latency and batch size for LLM serving workloads.
• 02Runs without GPUs after an initial profiling phase on target hardware.
• 03Supports tensor and pipeline parallelism combinations across several GPU node types.
• 04Accepts synthetic request generators and Azure LLM inference traces.
• 05Exports Chrome traces and logs metrics to wandb or local output directories.
• 06Extensible for testing new schedulers and optimizations.

Use cases

• Capacity planning to find the cheapest deployment meeting latency targets.
• Comparing scheduling policies under the same traffic trace.
• Researching speculative decoding or other serving optimizations before implementing them on real GPUs.
• Studying how model size and parallelism settings affect serving performance.

Quick start

1. 1

   Create a venv with Python 3.10

   python3.10 -m venv .venv
2. 2

   Activate the virtual environment

   source .venv/bin/activate
3. 3

   Install the dependencies

   python -m pip install -r requirements.txt
4. 4

   Run the simulator from the repository root

   python -m vidur.main

Things to know

• !Needs Python 3.10 and an initial GPU profiling phase for new models or SKUs.
• !Supported models and hardware are limited to the listed configurations.
• !Some features like prefix caching are only on the canary branch.

In collections

Topics

FAQ

What is vidur?

A high-fidelity simulator for LLM inference systems that helps study performance, plan deployments, and test scheduling ideas without GPUs. Vidur simulates how large language models serve requests on GPU clusters. It models metrics such as time to first token (TTFT), time per output token (TPOT), end-to-end request latency, and batch size under a chosen workload, model, and hardware configuration. The simulator is driven by a short initial profiling phase on real GPUs, after which experiments run without access to GPUs.

Is vidur free to use? What license does it have?

vidur is open source under the MIT license. Check the license terms before using it in commercial products.

Is vidur still maintained?

Updated within 6 months. The last commit to vidur was on Aug 24, 2026.

What can I use vidur for?

Capacity planning to find the cheapest deployment meeting latency targets. Comparing scheduling policies under the same traffic trace. Researching speculative decoding or other serving optimizations before implementing them on real GPUs. Studying how model size and parallelism settings affect serving performance.

How do I install vidur?

Create a venv with Python 3.10: python3.10 -m venv .venv Activate the virtual environment: source .venv/bin/activate Install the dependencies: python -m pip install -r requirements.txt Run the simulator from the repository root: python -m vidur.main

Where can I get vidur?

The source code and installation instructions are on GitHub: https://github.com/microsoft/vidur