Repolode
Submit
jundot

jundot/omlx

LLM inference server for Apple Silicon Macs with continuous batching, tiered RAM/SSD KV caching, and menu bar management.

Open-source alternative to:LM StudioOllama

View on GitHubWebsite
Stars
23,441
Forks
2,105
Open issues
577
Last commit
3 days ago

What is it?

oMLX is an LLM inference server built for Apple Silicon Macs on top of MLX and mlx-lm. It exposes OpenAI-compatible and Anthropic-compatible APIs at a local address, so existing clients and coding tools can connect to models running on the machine. It serves text LLMs, vision-language models, OCR models, embeddings, and rerankers from one server.

The server keeps KV cache in two tiers: frequently used blocks stay in RAM, and older blocks are written to SSD and restored when a matching prefix appears again, even after a restart. It also handles concurrent requests with continuous batching, and manages multiple loaded models with LRU eviction, pinning, per-model TTL, and a total memory limit.

A native SwiftUI menu bar app starts, stops, and monitors the server, and a web admin dashboard at /admin covers model downloads from Hugging Face, chat, benchmarks, and per-model settings. Setup is available through a macOS DMG, Homebrew, or a source install.

Key features

  • 01Serves OpenAI- and Anthropic-compatible chat, completion, embedding, and rerank endpoints.
  • 02Tiered KV cache with a RAM hot tier and an SSD cold tier that survives restarts.
  • 03Continuous batching for concurrent requests via mlx-lm's BatchGenerator.
  • 04Multi-model serving with LRU eviction, pinning, per-model TTL, and memory limits.
  • 05Native menu bar app plus a web admin dashboard for monitoring, downloads, and settings.
  • 06Automatic tool-call format detection for many model families, with MCP support.

Use cases

  • Running local coding assistants such as Claude Code against models on a Mac.
  • Serving a local OpenAI-compatible endpoint for apps that expect a cloud API.
  • Keeping frequently used models loaded while swapping heavier ones on demand.
  • Running embedding, reranking, or OCR models alongside chat models on one Mac.

Quick start

  1. 1

    Add the Homebrew tap

    brew tap jundot/omlx https://github.com/jundot/omlx
  2. 2

    Install oMLX with Homebrew

    brew install jundot/omlx/omlx
  3. 3

    Start a foreground server with your model directory

    omlx serve --model-dir ~/models

Things to know

  • Apple Silicon Macs only, requiring macOS 15.0 or later.
  • Some model families need native custom kernels, which require full Xcode with the Metal toolchain for source builds.

In collections

Run LLMs locallyOpen-source Mac apps

Topics

#apple-silicon#llm-inference#mlx#openai-api#kv-cache#local-llm#macos#inference-server#llm

FAQ

What is omlx?
LLM inference server for Apple Silicon Macs with continuous batching, tiered RAM/SSD KV caching, and menu bar management. oMLX is an LLM inference server built for Apple Silicon Macs on top of MLX and mlx-lm. It exposes OpenAI-compatible and Anthropic-compatible APIs at a local address, so existing clients and coding tools can connect to models running on the machine. It serves text LLMs, vision-language models, OCR models, embeddings, and rerankers from one server.
Is omlx free to use? What license does it have?
omlx is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is omlx still maintained?
Actively maintained. The last commit to omlx was on Nov 5, 2026.
What can I use omlx for?
Running local coding assistants such as Claude Code against models on a Mac. Serving a local OpenAI-compatible endpoint for apps that expect a cloud API. Keeping frequently used models loaded while swapping heavier ones on demand. Running embedding, reranking, or OCR models alongside chat models on one Mac.
How do I install omlx?
Add the Homebrew tap: brew tap jundot/omlx https://github.com/jundot/omlx Install oMLX with Homebrew: brew install jundot/omlx/omlx Start a foreground server with your model directory: omlx serve --model-dir ~/models
Where can I get omlx?
The source code and installation instructions are on GitHub: https://github.com/jundot/omlx

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects