michaelfeil/infinity

What is it?

Key features

• 01Serves embeddings, reranking, classification and typed decisions over a REST API.
• 02Runs inference with LibTorch in a Rust server.
• 03Uses all visible CUDA GPUs by default, with a full model replica on each.
• 04Provides SPLADE sparse embeddings and token-vector embeddings for late-interaction retrieval.
• 05Reports readiness after model loading and warmup via /health.

Use cases

• Hosting an embedding endpoint for semantic search or retrieval-augmented generation on NVIDIA GPUs.
• Adding a reranking stage to a search pipeline behind an HTTP API.
• Running named-entity recognition or sequence classification models as a service.
• Deploying a pinned container image for reproducible GPU inference.

Quick start

1. 1

   Run the Infinity v3 server in Docker (needs an NVIDIA GPU, SM80 or newer, and NVIDIA Container Toolkit)

   docker run --rm --gpus all -p 8000:8000 -v infinity-cache:/data ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.0 --model-id sentence-transformers/all-MiniLM-L6-v2 --torch-device cuda --dtype bfloat16 --hostname 0.0.0.0 --port 8000
2. 2

   Send an embedding request to the running server

   curl http://localhost:8000/v1/embeddings -H 'Content-Type: application/json' -d '{"model":"sentence-transformers/all-MiniLM-L6-v2","input":["A document about semantic search."]}'

Things to know

• !The v3 server uses a custom source-available license (Infinity Restricted MIT License 1.0), not a standard open-source license.
• !The documented Docker quick start requires an NVIDIA GPU with compute capability SM80 or newer.
• !Each GPU must fit the full model, including all experts for MoE checkpoints.

In collections

Topics

FAQ

What is infinity?

A Rust/LibTorch serving engine that exposes embeddings, reranking, classification and typed decisions over a REST API. Infinity is a model serving engine for text embeddings, reranking, classification and related models. Its v3 server is written in Rust and runs inference through LibTorch, the C++ interface of PyTorch. Clients call it over HTTP, using endpoints such as /v1/embeddings for dense vectors, /rerank for reranking, /v1/classify for sequence classification and reward scoring, and /embed_sparse for SPLADE sparse embeddings.

Is infinity free to use? What license does it have?

infinity is open source under the MIT license. Check the license terms before using it in commercial products.

Is infinity still maintained?

Actively maintained. The last commit to infinity was on Nov 7, 2026.

What can I use infinity for?

Hosting an embedding endpoint for semantic search or retrieval-augmented generation on NVIDIA GPUs. Adding a reranking stage to a search pipeline behind an HTTP API. Running named-entity recognition or sequence classification models as a service. Deploying a pinned container image for reproducible GPU inference.

How do I install infinity?

Run the Infinity v3 server in Docker (needs an NVIDIA GPU, SM80 or newer, and NVIDIA Container Toolkit): docker run --rm --gpus all -p 8000:8000 -v infinity-cache:/data ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.0 --model-id sentence-transformers/all-MiniLM-L6-v2 --torch-device cuda --dtype bfloat16 --hostname 0.0.0.0 --port 8000 Send an embedding request to the running server: curl http://localhost:8000/v1/embeddings -H 'Content-Type: application/json' -d '{"model":"sentence-transformers/all-MiniLM-L6-v2","input":["A document about semantic search."]}'

Where can I get infinity?

The source code and installation instructions are on GitHub: https://github.com/michaelfeil/infinity