Repolode
Submit
mlc-ai

mlc-ai/mlc-llm

A machine learning compiler and deployment engine that runs large language models natively across GPUs, browsers, and mobile devices with OpenAI-compatible APIs.

Open-source alternative to:OpenAI API

View on GitHubWebsite
Stars
23,261
Forks
2,162
Open issues
359
Last commit
15 days ago

What is it?

MLC LLM is a machine learning compiler and high-performance inference engine for large language models. It aims to let developers build, optimize, and deploy models natively on a wide range of hardware, rather than relying on a single vendor's stack.

The engine targets AMD, NVIDIA, Apple, and Intel GPUs on Linux, Windows, and macOS through backends such as Vulkan, ROCm, CUDA, and Metal. It also runs in web browsers via WebGPU and WASM, on iOS and iPadOS through Metal, and on Android through OpenCL on Adreno and Mali GPUs.

MLCEngine, the runtime, exposes an OpenAI-compatible API through a REST server, Python, JavaScript, iOS, and Android bindings, all sharing the same compiler and engine.

Key features

  • 01Runs LLMs across NVIDIA, AMD, Apple, and Intel GPUs via Vulkan, CUDA, ROCm, and Metal.
  • 02Runs models in web browsers with WebGPU and WASM.
  • 03Provides iOS and Android deployment support.
  • 04Offers an OpenAI-compatible API via REST server, Python, and JavaScript.
  • 05Built on a machine learning compiler that optimizes models for each target platform.

Use cases

  • Deploying a local LLM on a laptop or workstation regardless of GPU vendor.
  • Running language models directly in a web page without a backend server.
  • Shipping on-device LLM features in iOS or Android apps.
  • Replacing OpenAI API calls with a self-hosted, OpenAI-compatible endpoint.

Quick start

  1. 1

    Read the official quick start and installation guide

  2. 2

    Follow the documentation to get started with MLC LLM

Things to know

  • Installation and setup are documented only on an external site, not in the README itself.
  • Platform support varies by hardware, with some combinations such as Intel GPU on macOS or Apple GPUs on Linux and Windows unavailable.

Topics

#llm#machine-learning-compiler#inference#webgpu#on-device-ai#openai-compatible#gpu#language-model#machine-learning-compilation#tvm

FAQ

What is mlc-llm?
A machine learning compiler and deployment engine that runs large language models natively across GPUs, browsers, and mobile devices with OpenAI-compatible APIs. MLC LLM is a machine learning compiler and high-performance inference engine for large language models. It aims to let developers build, optimize, and deploy models natively on a wide range of hardware, rather than relying on a single vendor's stack.
Is mlc-llm free to use? What license does it have?
mlc-llm is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is mlc-llm still maintained?
Actively maintained. The last commit to mlc-llm was on Oct 6, 2026.
What can I use mlc-llm for?
Deploying a local LLM on a laptop or workstation regardless of GPU vendor. Running language models directly in a web page without a backend server. Shipping on-device LLM features in iOS or Android apps. Replacing OpenAI API calls with a self-hosted, OpenAI-compatible endpoint.
How do I install mlc-llm?
Read the official quick start and installation guide Follow the documentation to get started with MLC LLM
Where can I get mlc-llm?
The source code and installation instructions are on GitHub: https://github.com/mlc-ai/mlc-llm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects