Repolode
Submit
dphnAI

dphnAI/sonar

An LLM inference engine for Hugging Face-compatible models, built on vLLM, with OpenAI-compatible serving and distributed deployment.

View on GitHubWebsite
Stars
1,904
Forks
213
Open issues
55
Last commit
4 days ago

What is it?

Sonar is an inference engine for serving Hugging Face-compatible language and multimodal models. It covers continuous batching, paged KV-cache management, optimized kernels, quantization, speculative decoding, and distributed serving. The project is based on vLLM and adds further model and quantization formats, sampling methods, kernels, platforms, and deployment features.

The `aphrodite serve` command starts an HTTP server on port 2242 by default. It exposes OpenAI-compatible APIs along with health checks, Prometheus-style metrics, and an OpenAPI schema. Supported hardware includes NVIDIA CUDA, AMD ROCm, Intel XPU, CPU, Apple silicon Metal, and Google TPU, with install paths documented for each.

Support varies by model, device, data type, and quantization method, so the project's generated model and quantization matrices should be checked before deployment.

Key features

  • 01Continuous batching with paged KV-cache management and prefix caching
  • 02Tensor, pipeline, data, and expert parallelism, including multi-node setups without Ray
  • 03Speculative decoding with MTP, EAGLE, n-gram and other methods
  • 04Quantized weights and FP8 KV cache
  • 05LoRA adapter serving and image, audio, and video model support
  • 06OpenAI and Anthropic compatible APIs with structured output and tool calling

Use cases

  • Serving open-weight LLMs behind an OpenAI-compatible API in production
  • Running quantized models on limited GPU memory
  • Deploying multi-GPU or multi-node inference with parallelism
  • Hosting LoRA fine-tuned variants of a base model

Quick start

  1. 1

    Run the official install script from the README

  2. 2

    Serve a model with the OpenAI-compatible server

    aphrodite serve Qwen/Qwen3-0.6B --served-model-name qwen3

Things to know

  • Licensed under AGPL-3.0, a copyleft license that requires derivative network services to share source code.
  • Support varies by model, device, data type, and quantization method, so check the model and quantization matrices before deployment.

In collections

Run LLMs locallyRun and train AI on GPUs

Topics

#llm-inference#inference-engine#openai-compatible#quantization#speculative-decoding#lora#gpu#distributed-inference#api-rest#cuda

FAQ

What is sonar?
An LLM inference engine for Hugging Face-compatible models, built on vLLM, with OpenAI-compatible serving and distributed deployment. Sonar is an inference engine for serving Hugging Face-compatible language and multimodal models. It covers continuous batching, paged KV-cache management, optimized kernels, quantization, speculative decoding, and distributed serving. The project is based on vLLM and adds further model and quantization formats, sampling methods, kernels, platforms, and deployment features.
Is sonar free to use? What license does it have?
sonar is open source under the AGPL-3.0 license. Check the license terms before using it in commercial products.
Is sonar still maintained?
Actively maintained. The last commit to sonar was on Nov 3, 2026.
What can I use sonar for?
Serving open-weight LLMs behind an OpenAI-compatible API in production Running quantized models on limited GPU memory Deploying multi-GPU or multi-node inference with parallelism Hosting LoRA fine-tuned variants of a base model
How do I install sonar?
Run the official install script from the README Serve a model with the OpenAI-compatible server: aphrodite serve Qwen/Qwen3-0.6B --served-model-name qwen3
Where can I get sonar?
The source code and installation instructions are on GitHub: https://github.com/dphnAI/sonar

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects