Repolode
Submit
basetenlabs

basetenlabs/truss

CLI and Python toolkit for packaging and deploying ML models to Baseten or your own infrastructure as production model servers.

View on GitHubWebsite
Stars
1,228
Forks
142
Open issues
90
Last commit
18 hours ago

What is it?

Truss packages a model's serving logic, weights, and dependencies into a container-ready model server, then deploys it to Baseten or your own infrastructure. It is driven by a Python model class for custom code and a config.yaml that declares hardware, such as GPU type, and build options. Developers can iterate with a live-reload development mode, while Docker and Kubernetes configuration is handled for them.

The project supports models from common open-source frameworks, including vLLM, SGLang, TensorRT-LLM, transformers, diffusers, PyTorch, and TensorFlow. For LLMs, a config can build a TensorRT-LLM engine with options such as FP8 quantization and maximum sequence length, and the resulting deployment exposes an OpenAI-compatible API. Production features include GPU configuration, secrets, caching, and autoscaling on Baseten.

It suits teams that want to turn a Hugging Face or custom PyTorch model into a hosted endpoint without writing serving infrastructure. The repository also includes a JSON schema for config.yaml that enables editor autocompletion.

Key features

  • 01Packages model code, weights, and dependencies into a model server
  • 02Live-reload development mode for faster iteration
  • 03Builds TensorRT-LLM engines with FP8 quantization from a YAML config
  • 04Serves OpenAI-compatible endpoints for LLM deployments
  • 05Supports vLLM, SGLang, transformers, diffusers, PyTorch, and TensorFlow models
  • 06Provides GPU configuration, secrets, caching, and autoscaling on Baseten

Use cases

  • Deploying a Hugging Face LLM as a hosted API endpoint without writing Dockerfiles
  • Serving a custom PyTorch or diffusion model with GPU acceleration in production
  • Replacing hand-built model serving stacks with a config-driven workflow

Quick start

  1. 1

    Install Truss with pip

    pip install --upgrade truss
  2. 2

    Log in to Baseten

    uvx truss login
  3. 3

    Scaffold a new project

    uvx truss init qwen-2.5-3b && cd qwen-2.5-3b
  4. 4

    Deploy the model to Baseten

    uvx truss push

Things to know

  • Deploying the quickstart requires a Baseten account and API key.
  • The quickstart's TRT-LLM deployment needs an L4 GPU on Baseten, and --watch mode is not supported for TRT-LLM.

In collections

Run and train AI on GPUs

Topics

#model-serving#machine-learning#inference-server#llm#gpu#deployment#python#pytorch#artificial-intelligence#easy-to-use

FAQ

What is truss?
CLI and Python toolkit for packaging and deploying ML models to Baseten or your own infrastructure as production model servers. Truss packages a model's serving logic, weights, and dependencies into a container-ready model server, then deploys it to Baseten or your own infrastructure. It is driven by a Python model class for custom code and a config.yaml that declares hardware, such as GPU type, and build options. Developers can iterate with a live-reload development mode, while Docker and Kubernetes configuration is handled for them.
Is truss free to use? What license does it have?
truss is open source under the MIT license. Check the license terms before using it in commercial products.
Is truss still maintained?
Actively maintained. The last commit to truss was on Nov 5, 2026.
What can I use truss for?
Deploying a Hugging Face LLM as a hosted API endpoint without writing Dockerfiles Serving a custom PyTorch or diffusion model with GPU acceleration in production Replacing hand-built model serving stacks with a config-driven workflow
How do I install truss?
Install Truss with pip: pip install --upgrade truss Log in to Baseten: uvx truss login Scaffold a new project: uvx truss init qwen-2.5-3b && cd qwen-2.5-3b Deploy the model to Baseten: uvx truss push
Where can I get truss?
The source code and installation instructions are on GitHub: https://github.com/basetenlabs/truss

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects