Repolode
Submit
jjang-ai

jjang-ai/vmlx

Self-hosted MLX inference server for Apple Silicon with OpenAI, Anthropic and Ollama compatible APIs for LLMs, VLMs and image generation.

Open-source alternative to:LM StudioOllama

View on GitHubWebsite
Stars
899
Forks
97
Open issues
63
Last commit
yesterday

What is it?

vMLX is an inference server for running large language models, vision-language models and image generation models locally on Mac computers with Apple Silicon. It exposes OpenAI-compatible, Anthropic-compatible and Ollama-compatible HTTP endpoints, so existing SDKs and tools such as the OpenAI Python client, the Anthropic SDK or the Ollama CLI can point at a local server without third-party API keys. Models are loaded from Hugging Face repositories or local paths, with mlx-community models as the typical starting point.

The server includes several caching layers, including prefix caching, paged KV cache, KV quantization and optional disk cache, which reuse prompt state for repeated or follow-up messages. It also supports continuous batching, tool calling and reasoning parsers for many model families, speculative decoding, and multi-Mac pipeline parallelism for models too large for one machine. Image generation and editing with Flux and Qwen Image Edit, plus embeddings, reranking, Whisper speech-to-text and Kokoro text-to-speech, are available through the same server.

The project also ships JANG, an adaptive mixed-precision quantization format with conversion tooling, and MLX Studio, a macOS desktop app with chat, server management and developer tools. It is installable from PyPI as `vmlx`.

Key features

  • 01Serves OpenAI, Anthropic and Ollama compatible chat, completion, embedding and rerank endpoints.
  • 02Multi-layer prefix, paged and disk KV caching with optional q4/q8 cache quantization.
  • 03Continuous batching and speculative or prompt lookup decoding for faster generation.
  • 04Auto-detected tool-calling and reasoning (thinking) parsers across many model families.
  • 05Image generation and editing with Flux and Qwen Image Edit models.
  • 06Distributed pipeline parallelism across multiple Macs for larger models.

Use cases

  • Running open-weight chat models privately on a Mac without sending data to cloud APIs.
  • Pointing existing OpenAI or Anthropic SDK code at a local model during development.
  • Serving a local model to several tools at once through one API gateway.
  • Generating or editing images locally on Apple Silicon hardware.

Quick start

  1. 1

    Install uv with Homebrew

    brew install uv
  2. 2

    Install vmlx from PyPI as a tool

    uv tool install vmlx
  3. 3

    Start the server with a model from mlx-community

    vmlx serve mlx-community/Qwen3-8B-4bit

Things to know

  • Runs only on Apple Silicon Macs, so it does not work on Linux, Windows or Intel Macs.
  • Requires Python 3.10 or newer.

In collections

Run LLMs locallyOpen-source Mac apps

Topics

#mlx#apple-silicon#llm#inference-server#openai-api#local-llm#kv-cache#quantization#anthropic-api#kvcache-compression

FAQ

What is vmlx?
Self-hosted MLX inference server for Apple Silicon with OpenAI, Anthropic and Ollama compatible APIs for LLMs, VLMs and image generation. vMLX is an inference server for running large language models, vision-language models and image generation models locally on Mac computers with Apple Silicon. It exposes OpenAI-compatible, Anthropic-compatible and Ollama-compatible HTTP endpoints, so existing SDKs and tools such as the OpenAI Python client, the Anthropic SDK or the Ollama CLI can point at a local server without third-party API keys. Models are loaded from Hugging Face repositories or local paths, with mlx-community models as the typical starting point.
Is vmlx free to use? What license does it have?
vmlx is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is vmlx still maintained?
Actively maintained. The last commit to vmlx was on Oct 21, 2026.
What can I use vmlx for?
Running open-weight chat models privately on a Mac without sending data to cloud APIs. Pointing existing OpenAI or Anthropic SDK code at a local model during development. Serving a local model to several tools at once through one API gateway. Generating or editing images locally on Apple Silicon hardware.
How do I install vmlx?
Install uv with Homebrew: brew install uv Install vmlx from PyPI as a tool: uv tool install vmlx Start the server with a model from mlx-community: vmlx serve mlx-community/Qwen3-8B-4bit
Where can I get vmlx?
The source code and installation instructions are on GitHub: https://github.com/jjang-ai/vmlx

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects