Repolode
Submit
huggingface

huggingface/text-embeddings-inference

A Rust-based server for deploying and serving open-source text embedding and reranker models over HTTP and gRPC.

View on GitHubWebsite
Stars
5,104
Forks
450
Open issues
217
Last commit
22 hours ago

What is it?

Text Embeddings Inference (TEI) is a toolkit for deploying and serving open-source text embedding and sequence classification models. It exposes endpoints for generating embeddings, reranking query-document pairs with cross-encoder models, and running sequence classification, with an OpenAPI/Swagger reference for the REST API.

The server supports models based on architectures such as BERT, XLM-RoBERTa, Qwen2/Qwen3, Mistral, Gemma3, ModernBERT and NomicBERT, and loads weights in Safetensors or ONNX format. Requests are batched dynamically by token count, and the server can run on CPU, NVIDIA GPUs via CUDA, Apple Silicon with Metal, and AMD Instinct GPUs via ROCm, using prebuilt Docker images or local builds with Cargo.

For operations, TEI provides OpenTelemetry distributed tracing, Prometheus metrics, JSON logs, optional API key authorization, and support for air-gapped deployments where model files are mounted from a local volume.

Key features

  • 01Serves embedding, reranking and sequence classification models over HTTP REST and gRPC.
  • 02Dynamic batching based on token count to make use of available hardware.
  • 03Runs on CPU, NVIDIA CUDA GPUs, Apple Silicon Metal and AMD ROCm GPUs.
  • 04Loads Safetensors and ONNX weights and pulls models from the Hugging Face Hub or a local directory.
  • 05Provides OpenTelemetry tracing, Prometheus metrics and optional API key authorization.

Use cases

  • Hosting an embedding endpoint for a retrieval-augmented generation (RAG) pipeline.
  • Adding a cross-encoder reranking step to improve search result ordering.
  • Running embedding models in air-gapped or private infrastructure with local weights.
  • Serving embedding models on Apple Silicon Macs for local development.

Quick start

  1. 1

    Run the Docker container with a model (needs NVIDIA Container Toolkit for GPU)

    docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model
  2. 2

    Send an embedding request to the local server

    curl 127.0.0.1:8080/embed -X POST -d '{"inputs":"What is Deep Learning?"}' -H 'Content-Type: application/json'

Things to know

  • GPU use requires the NVIDIA Container Toolkit and CUDA 12.2 or higher drivers.
  • Supported model architectures are limited to those listed in the README.

In collections

Run and train AI on GPUs

Topics

#embeddings#inference#rag#rust#reranker#vector-search#gpu#onnx#ai#huggingface

FAQ

What is text-embeddings-inference?
A Rust-based server for deploying and serving open-source text embedding and reranker models over HTTP and gRPC. Text Embeddings Inference (TEI) is a toolkit for deploying and serving open-source text embedding and sequence classification models. It exposes endpoints for generating embeddings, reranking query-document pairs with cross-encoder models, and running sequence classification, with an OpenAPI/Swagger reference for the REST API.
Is text-embeddings-inference free to use? What license does it have?
text-embeddings-inference is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is text-embeddings-inference still maintained?
Actively maintained. The last commit to text-embeddings-inference was on Nov 5, 2026.
What can I use text-embeddings-inference for?
Hosting an embedding endpoint for a retrieval-augmented generation (RAG) pipeline. Adding a cross-encoder reranking step to improve search result ordering. Running embedding models in air-gapped or private infrastructure with local weights. Serving embedding models on Apple Silicon Macs for local development.
How do I install text-embeddings-inference?
Run the Docker container with a model (needs NVIDIA Container Toolkit for GPU): docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model Send an embedding request to the local server: curl 127.0.0.1:8080/embed -X POST -d '{"inputs":"What is Deep Learning?"}' -H 'Content-Type: application/json'
Where can I get text-embeddings-inference?
The source code and installation instructions are on GitHub: https://github.com/huggingface/text-embeddings-inference

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects