Repolode
Submit
mlc-ai

mlc-ai/web-llm

A TypeScript engine that runs LLM inference entirely in the browser using WebGPU, with an OpenAI-compatible API.

View on GitHubWebsite
Stars
19,291
Forks
1,401
Open issues
154
Last commit
10 days ago

What is it?

WebLLM is an in-browser inference engine for large language models. Inference runs entirely on the client with no server involved, and it uses WebGPU for hardware acceleration. It is distributed as an npm package (@mlc-ai/web-llm) and can also be loaded directly from a CDN.

The engine exposes the same chat completion interface as the OpenAI API, so existing OpenAI-style code can target open models running locally in the browser. It supports streaming responses, JSON-mode output, seeded generation, and logit-level control. Function calling is listed as work in progress.

Models are used in the MLC format, and the project supports families such as Llama 3, Phi 3, Gemma, Mistral, and Qwen. Developers can run the engine in a dedicated Web Worker or Service Worker, or inside a Chrome extension, and can configure browser cache backends such as Cache API, IndexedDB, and OPFS.

Key features

  • 01Runs LLM inference in the browser with WebGPU acceleration and no server.
  • 02Provides an OpenAI-compatible chat completions API with streaming and JSON mode.
  • 03Supports models such as Llama 3, Phi 3, Gemma, Mistral, and Qwen in MLC format.
  • 04Can run in Web Workers, Service Workers, or Chrome extensions.
  • 05Offers SRI-based integrity verification for downloaded model artifacts.

Use cases

  • Building a privacy-preserving chatbot where prompts never leave the user's device.
  • Adding a local AI assistant to a web app using an OpenAI-style client.
  • Developing Chrome extensions that call a language model locally.
  • Prototyping browser-based LLM features without running backend inference servers.

Quick start

  1. 1

    Install the package with npm

    npm install @mlc-ai/web-llm
  2. 2

    Import the engine in your code

    import { CreateMLCEngine } from "@mlc-ai/web-llm";
  3. 3

    Create the engine with a model ID (first load downloads the model)

    const engine = await CreateMLCEngine("Llama-3.1-8B-Instruct-q4f32_1-MLC", { initProgressCallback });

Things to know

  • Requires a browser with WebGPU support.
  • The first model load downloads large model files, which can take a long time.

Topics

#llm#webgpu#in-browser#openai-compatible#inference#typescript#webassembly#chatgpt#deep-learning#language-model

FAQ

What is web-llm?
A TypeScript engine that runs LLM inference entirely in the browser using WebGPU, with an OpenAI-compatible API. WebLLM is an in-browser inference engine for large language models. Inference runs entirely on the client with no server involved, and it uses WebGPU for hardware acceleration. It is distributed as an npm package (@mlc-ai/web-llm) and can also be loaded directly from a CDN.
Is web-llm free to use? What license does it have?
web-llm is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is web-llm still maintained?
Actively maintained. The last commit to web-llm was on Oct 11, 2026.
What can I use web-llm for?
Building a privacy-preserving chatbot where prompts never leave the user's device. Adding a local AI assistant to a web app using an OpenAI-style client. Developing Chrome extensions that call a language model locally. Prototyping browser-based LLM features without running backend inference servers.
How do I install web-llm?
Install the package with npm: npm install @mlc-ai/web-llm Import the engine in your code: import { CreateMLCEngine } from "@mlc-ai/web-llm"; Create the engine with a model ID (first load downloads the model): const engine = await CreateMLCEngine("Llama-3.1-8B-Instruct-q4f32_1-MLC", { initProgressCallback });
Where can I get web-llm?
The source code and installation instructions are on GitHub: https://github.com/mlc-ai/web-llm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects