Repolode
Submit
mosecorg

mosecorg/mosec

A Python ML model serving framework with a Rust web layer, offering dynamic batching and pipelined CPU/GPU stages for online inference APIs.

View on GitHubWebsite
Stars
902
Forks
75
Open issues
13
Last commit
5 days ago

What is it?

Mosec is a model serving framework for turning trained machine learning models into online HTTP services. The web layer and task coordination are written in Rust, while users define their service logic in Python, so the same inference code used for offline testing can be served without rewriting it for a specific ML framework.

The framework aggregates concurrent requests into batches according to a configured maximum batch size and wait time, then distributes results back to each client. Multi-stage pipelines let separate processes handle preprocessing on CPU and inference on GPU, and worker counts can be tuned per stage. Serving features include model warmup, graceful shutdown, Prometheus metrics, and OpenAPI documentation generation, and it is designed to run under Kubernetes or other container orchestrators.

Mosec is installable from PyPI and conda-forge for Linux and macOS. Its examples cover PyTorch, JAX, and Stable Diffusion models, as well as OpenAI-compatible embedding and reranking services.

Key features

  • 01Dynamic batching that groups concurrent requests by max batch size or wait time.
  • 02Multi-stage pipelines that run CPU and GPU work in separate processes.
  • 03Rust-based web layer with a pure Python interface for writing workers.
  • 04Model warmup, graceful shutdown, and Prometheus metrics for production use.
  • 05OpenAPI documentation generated for served endpoints.
  • 06Msgpack and JSON serialization, with support for shared-memory IPC.

Use cases

  • Serving a PyTorch or Stable Diffusion model as an HTTP API with batched GPU inference.
  • Building multi-stage inference services that split preprocessing and model execution.
  • Deploying OpenAI-compatible embedding or reranking endpoints in containers on Kubernetes.

Quick start

  1. 1

    Install the mosec package from PyPI

    pip install -U mosec
  2. 2

    Install diffusers and transformers prerequisites (only needed for the stable diffusion example)

    pip install --upgrade diffusers[torch] transformers
  3. 3

    Run the stable diffusion example server from the project root

    python examples/stable_diffusion/server.py --log-level debug --timeout 30000

Things to know

  • Installation via pip is supported for Linux or macOS only.
  • Python 3.7 or above is required.

In collections

Run and train AI on GPUs

Topics

#machine-learning#model-serving#inference#dynamic-batching#gpu#python#rust#mlops#cv#deep-learning

FAQ

What is mosec?
A Python ML model serving framework with a Rust web layer, offering dynamic batching and pipelined CPU/GPU stages for online inference APIs. Mosec is a model serving framework for turning trained machine learning models into online HTTP services. The web layer and task coordination are written in Rust, while users define their service logic in Python, so the same inference code used for offline testing can be served without rewriting it for a specific ML framework.
Is mosec free to use? What license does it have?
mosec is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is mosec still maintained?
Actively maintained. The last commit to mosec was on Nov 2, 2026.
What can I use mosec for?
Serving a PyTorch or Stable Diffusion model as an HTTP API with batched GPU inference. Building multi-stage inference services that split preprocessing and model execution. Deploying OpenAI-compatible embedding or reranking endpoints in containers on Kubernetes.
How do I install mosec?
Install the mosec package from PyPI: pip install -U mosec Install diffusers and transformers prerequisites (only needed for the stable diffusion example): pip install --upgrade diffusers[torch] transformers Run the stable diffusion example server from the project root: python examples/stable_diffusion/server.py --log-level debug --timeout 30000
Where can I get mosec?
The source code and installation instructions are on GitHub: https://github.com/mosecorg/mosec

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects