Repolode
Submit
bentoml

bentoml/BentoML

A Python library for building and serving AI model inference APIs, multi-model pipelines and LLM apps, with Docker and Kubernetes deployment.

View on GitHubWebsite
Stars
8,914
Forks
1,049
Open issues
239
Last commit
2 days ago

What is it?

BentoML is a Python library for turning AI and machine learning models into online serving systems. Any inference script can be exposed as a REST API with a few lines of code and standard Python type hints. Services are defined in a service.py file, and environment and dependency settings are managed through a configuration file.

The library includes serving optimizations such as dynamic batching, model parallelism, multi-stage pipelines and multi-model inference graph orchestration. It can be run locally for development and debugging, then packaged into a standardized artifact called a Bento and containerized as a Docker image. Deployment to Kubernetes is also supported.

The project covers LLMs, image generation, embeddings, audio and computer vision models, and its examples repositories provide ready-made services for many of them.

Key features

  • 01Turns Python inference code into REST API servers using type hints.
  • 02Supports dynamic batching, model parallelism and multi-model pipelines.
  • 03Packages services into Bento artifacts and generates Docker images.
  • 04Deploys to Docker containers or Kubernetes.
  • 05Provides a Python HTTP client for calling running services.

Use cases

  • Serving a Hugging Face or custom PyTorch model behind an HTTP API.
  • Running LLM inference services with batching and GPU utilization.
  • Composing several models into one inference pipeline.
  • Packaging ML services into Docker images for production deployment.

Quick start

  1. 1

    Install BentoML with pip

    pip install -U bentoml
  2. 2

    Install the example's extra dependencies (PyTorch and Transformers) for local runs

    pip install torch transformers
  3. 3

    Run the service locally at http://localhost:3000

    bentoml serve

Things to know

  • Requires Python 3.9 or newer.
  • The quick-start example downloads and runs large ML models such as PyTorch and Transformers, which are heavy dependencies.

Topics

#model-serving#inference#llm#mlops#machine-learning#docker#kubernetes#ai-inference#deep-learning#generative-ai

FAQ

What is BentoML?
A Python library for building and serving AI model inference APIs, multi-model pipelines and LLM apps, with Docker and Kubernetes deployment. BentoML is a Python library for turning AI and machine learning models into online serving systems. Any inference script can be exposed as a REST API with a few lines of code and standard Python type hints. Services are defined in a service.py file, and environment and dependency settings are managed through a configuration file.
Is BentoML free to use? What license does it have?
BentoML is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is BentoML still maintained?
Actively maintained. The last commit to BentoML was on Oct 20, 2026.
What can I use BentoML for?
Serving a Hugging Face or custom PyTorch model behind an HTTP API. Running LLM inference services with batching and GPU utilization. Composing several models into one inference pipeline. Packaging ML services into Docker images for production deployment.
How do I install BentoML?
Install BentoML with pip: pip install -U bentoml Install the example's extra dependencies (PyTorch and Transformers) for local runs: pip install torch transformers Run the service locally at http://localhost:3000: bentoml serve
Where can I get BentoML?
The source code and installation instructions are on GitHub: https://github.com/bentoml/BentoML

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects