Repolode
Submit
deepjavalibrary

deepjavalibrary/djl-serving

A Java-based, standalone model serving solution that exposes PyTorch, TensorFlow, ONNX and other deep learning models over HTTP endpoints.

View on GitHub
Stars
256
Forks
100
Open issues
16
Last commit
5 days ago

What is it?

DJL Serving is a standalone model serving system built on the DJL deep learning library for Java. It takes a single model, several models, or a workflow and makes them callable through an HTTP endpoint. Out of the box it serves PyTorch TorchScript models, TensorFlow SavedModel bundles, CPU-based ONNX models, and Python script models, with extensions available for XGBoost, LightGBM, Sentencepiece and fastText/BlazingText.

The server runs multithreaded inference inside a single JVM and can scale worker threads up or down with load. It also supports dynamic batching, loading multiple versions of a model behind one endpoint, and serving models from different engines at the same time. Separate inference and management REST APIs handle running models and adding, removing or scaling them at runtime.

DJL Serving can be installed through Homebrew, a Debian package, a zip archive on Windows, or run as a Docker container. Plugins allow custom extensions to be added.

Key features

  • 01Serves PyTorch, TensorFlow, ONNX and Python script models out of the box.
  • 02Runs multithreaded inference in a single JVM with auto-scaling worker threads.
  • 03Supports dynamic batching to improve throughput.
  • 04Allows multiple model versions on one endpoint and models from different engines together.
  • 05Provides separate REST APIs for inference and model management.
  • 06Extensible through a plugin system.

Use cases

  • Deploying trained deep learning models as HTTP inference services from a Java stack.
  • Serving several models or versions behind one endpoint with runtime scaling.
  • Hosting Python-based model scripts alongside TorchScript or ONNX models.

Quick start

  1. 1

    Install on macOS with Homebrew

    brew install djl-serving
  2. 2

    Or run it in Docker, serving on port 8080

    docker run -itd -p 8080:8080 deepjavalibrary/djl-serving

Things to know

  • ONNX models are supported on CPU only.
  • XGBoost, LightGBM, Sentencepiece and fastText models need extra extensions installed.
  • By default the server only accepts connections from localhost and needs configuration for remote access.

Topics

#deep-learning#model-serving#inference#java#pytorch#onnx#machine-learning-deployment#deployment#djl#serving

FAQ

What is djl-serving?
A Java-based, standalone model serving solution that exposes PyTorch, TensorFlow, ONNX and other deep learning models over HTTP endpoints. DJL Serving is a standalone model serving system built on the DJL deep learning library for Java. It takes a single model, several models, or a workflow and makes them callable through an HTTP endpoint. Out of the box it serves PyTorch TorchScript models, TensorFlow SavedModel bundles, CPU-based ONNX models, and Python script models, with extensions available for XGBoost, LightGBM, Sentencepiece and fastText/BlazingText.
Is djl-serving free to use? What license does it have?
djl-serving is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is djl-serving still maintained?
Actively maintained. The last commit to djl-serving was on Nov 2, 2026.
What can I use djl-serving for?
Deploying trained deep learning models as HTTP inference services from a Java stack. Serving several models or versions behind one endpoint with runtime scaling. Hosting Python-based model scripts alongside TorchScript or ONNX models.
How do I install djl-serving?
Install on macOS with Homebrew: brew install djl-serving Or run it in Docker, serving on port 8080: docker run -itd -p 8080:8080 deepjavalibrary/djl-serving
Where can I get djl-serving?
The source code and installation instructions are on GitHub: https://github.com/deepjavalibrary/djl-serving

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects