Repolode
Submit
openvinotoolkit

openvinotoolkit/model_server

A C++ inference server that serves OpenVINO-optimized generative AI and classic deep learning models over OpenAI-compatible and KServe APIs.

View on GitHubWebsite
Stars
952
Forks
284
Open issues
161
Last commit
6 hours ago

What is it?

OpenVINO Model Server (OVMS) is a C++ inference server that exposes machine learning models over standard network APIs. It covers LLMs, vision language models, embeddings, reranking, image generation and audio through OpenAI-compatible endpoints, and classic models such as object detection, classification and OCR through KServe APIs.

It runs on Docker, bare metal on Linux or Windows, and Kubernetes or OpenShift. Models can be loaded from local storage, S3, GCS, Azure Blob or the Hugging Face Hub, with versioning and hot reload. Inference runs on Intel CPU, integrated and discrete GPU, and NPU through OpenVINO.

The server supports TensorFlow, ONNX, PaddlePaddle and OpenVINO IR model formats, along with MediaPipe graphs and Python execution nodes for building pipelines. Prometheus-compatible metrics and a C API for embedding in native applications are also provided.

Key features

  • 01OpenAI-compatible endpoints for chat, embeddings, image generation and audio
  • 02KServe API for classic model inference
  • 03Continuous batching, streaming and speculative decoding for LLMs
  • 04Model versioning and hot reload from local, cloud or Hugging Face storage
  • 05MediaPipe graphs and Python nodes for multi-step pipelines
  • 06Prometheus-compatible metrics

Use cases

  • Serving a local LLM behind an OpenAI-compatible API on Intel hardware
  • Deploying classic vision models such as detection or OCR on Kubernetes
  • Running RAG components like embeddings and reranking models on CPU or GPU
  • Embedding model inference into a native C++ application via the C API

Quick start

  1. 1

    Pull the latest OpenVINO Model Server Docker image

    docker pull openvino/model_server:latest
  2. 2

    Run the server with an LLM using the Docker command in the Serve an LLM section of the README

Things to know

  • Officially supported only on Linux and Windows, with no macOS listed.
  • Acceleration is aimed at Intel hardware (CPU, GPU, NPU) through OpenVINO.

Topics

#inference-server#openvino#llm#model-serving#kubernetes#openai-api#intel#ai#cloud#dag

FAQ

What is model_server?
A C++ inference server that serves OpenVINO-optimized generative AI and classic deep learning models over OpenAI-compatible and KServe APIs. OpenVINO Model Server (OVMS) is a C++ inference server that exposes machine learning models over standard network APIs. It covers LLMs, vision language models, embeddings, reranking, image generation and audio through OpenAI-compatible endpoints, and classic models such as object detection, classification and OCR through KServe APIs.
Is model_server free to use? What license does it have?
model_server is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is model_server still maintained?
Actively maintained. The last commit to model_server was on Oct 21, 2026.
What can I use model_server for?
Serving a local LLM behind an OpenAI-compatible API on Intel hardware Deploying classic vision models such as detection or OCR on Kubernetes Running RAG components like embeddings and reranking models on CPU or GPU Embedding model inference into a native C++ application via the C API
How do I install model_server?
Pull the latest OpenVINO Model Server Docker image: docker pull openvino/model_server:latest Run the server with an LLM using the Docker command in the Serve an LLM section of the README
Where can I get model_server?
The source code and installation instructions are on GitHub: https://github.com/openvinotoolkit/model_server

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects