Repolode
Submit
tensorflow

tensorflow/serving

A high-performance system for serving machine learning models in production, with versioned gRPC and HTTP inference endpoints.

View on GitHubWebsite
Stars
6,365
Forks
2,204
Open issues
82
Last commit
17 days ago

What is it?

TensorFlow Serving handles the inference side of machine learning: it takes trained models, loads and manages their lifetimes, and gives clients versioned access to them. It works out of the box with TensorFlow SavedModels and can be extended to serve other model types and data such as embeddings, vocabularies and feature transformations.

The system can host multiple models or several versions of one model at the same time. New versions can be rolled out without changing client code, which makes canary releases and A/B tests of experimental models possible. A batching scheduler groups individual requests into batches for joint execution on GPU, with configurable latency controls.

The service exposes both gRPC and REST inference endpoints. It is distributed as a Docker image, and the documentation covers Kubernetes deployment, configuration, and building from source.

Key features

  • 01Serves multiple models or multiple versions of a model at once
  • 02Provides both gRPC and HTTP/REST inference endpoints
  • 03Lets new model versions deploy without client code changes
  • 04Supports canary releases and A/B testing of models
  • 05Batches requests for GPU execution with configurable latency controls

Use cases

  • Deploying TensorFlow models behind an inference API in production
  • Rolling out a new model version gradually while keeping the same endpoint
  • Running A/B tests between experimental and current models
  • Serving non-TensorFlow models or embeddings through a custom servable

Quick start

  1. 1

    Download the TensorFlow Serving Docker image

    docker pull tensorflow/serving
  2. 2

    Clone the repo to get demo models

    git clone https://github.com/tensorflow/serving
  3. 3

    Start the container with the demo model and REST port open

    docker run -t --rm -p 8501:8501 -v "$TESTDATA/saved_model_half_plus_two_cpu:/models/half_plus_two" -e MODEL_NAME=half_plus_two tensorflow/serving &
  4. 4

    Query the model using the predict API

    curl -d '{"instances": [1.0, 2.0, 5.0]}' -X POST http://localhost:8501/v1/models/half_plus_two:predict

Things to know

  • Docker is the recommended setup; running without Docker is not recommended by the README.
  • You must export your model as a TensorFlow SavedModel to serve it out of the box.

In collections

Run and train AI on GPUsMicroservice frameworks & service infrastructure

Topics

#model-serving#inference#tensorflow#machine-learning#grpc#gpu#kubernetes#deep-learning#cpp#deep-neural-networks

FAQ

What is serving?
A high-performance system for serving machine learning models in production, with versioned gRPC and HTTP inference endpoints. TensorFlow Serving handles the inference side of machine learning: it takes trained models, loads and manages their lifetimes, and gives clients versioned access to them. It works out of the box with TensorFlow SavedModels and can be extended to serve other model types and data such as embeddings, vocabularies and feature transformations.
Is serving free to use? What license does it have?
serving is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is serving still maintained?
Actively maintained. The last commit to serving was on Oct 21, 2026.
What can I use serving for?
Deploying TensorFlow models behind an inference API in production Rolling out a new model version gradually while keeping the same endpoint Running A/B tests between experimental and current models Serving non-TensorFlow models or embeddings through a custom servable
How do I install serving?
Download the TensorFlow Serving Docker image: docker pull tensorflow/serving Clone the repo to get demo models: git clone https://github.com/tensorflow/serving Start the container with the demo model and REST port open: docker run -t --rm -p 8501:8501 -v "$TESTDATA/saved_model_half_plus_two_cpu:/models/half_plus_two" -e MODEL_NAME=half_plus_two tensorflow/serving & Query the model using the predict API: curl -d '{"instances": [1.0, 2.0, 5.0]}' -X POST http://localhost:8501/v1/models/half_plus_two:predict
Where can I get serving?
The source code and installation instructions are on GitHub: https://github.com/tensorflow/serving

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects