Repolode
Submit
google-ai-edge

google-ai-edge/LiteRT-LM

Google's open-source framework for running large language models on edge devices across Android, iOS, Web, desktop and Raspberry Pi.

View on GitHubWebsite
Stars
6,710
Forks
764
Open issues
710
Last commit
3 hours ago

What is it?

LiteRT-LM is an orchestration layer for running LLMs on LiteRT, Google's on-device inference runtime. It targets developers who want to run models such as Gemma, Llama, Phi-4 and Qwen locally on phones, browsers, desktops and IoT boards instead of calling a cloud API.

The project provides a CLI for running and importing models from Hugging Face, and an OpenAI-compatible server endpoint (`litert-lm serve`) that also serves multimodal embedding models. Language bindings are offered for Python, Kotlin, Swift, JavaScript and C++, with Flutter available through community support. Hardware acceleration covers GPU and NPU backends, and the project includes tool use (function calling) and vision and audio inputs.

Google states that LiteRT-LM powers on-device GenAI features in Chrome, Chromebook Plus and Pixel Watch, and the repository links to the Google AI Edge Gallery app for trying models on a device.

Key features

  • 01Runs on Android, iOS, Web, desktop and Raspberry Pi.
  • 02Uses GPU and NPU accelerators for inference.
  • 03Supports vision and audio inputs alongside text.
  • 04Provides function calling for agentic workflows.
  • 05Offers an OpenAI-compatible server with embeddings endpoint.
  • 06Provides Python, Kotlin, Swift, JavaScript and C++ APIs.

Use cases

  • Building an Android or iOS app that runs a chat model offline.
  • Running a local LLM on a laptop or Raspberry Pi from the command line.
  • Adding on-device function calling to an app without a cloud backend.
  • Serving local embedding models through an OpenAI-style API.

Quick start

  1. 1

    Install the LiteRT-LM CLI with uv

    uv tool install litert-lm
  2. 2

    Run the Gemma 3n model from Hugging Face in your terminal

    litert-lm run \ --from-huggingface-repo=google/gemma-3n-E2B-it-litert-lm \ gemma-3n-E2B-it-int4 \ --prompt="What is the capital of France?"

Things to know

  • Swift and JavaScript APIs are in early preview.
  • Flutter support is community-maintained rather than official.

In collections

Android apps & phone tools

Topics

#edge-ai#on-device-ai#llm#inference#gemma#mobile#cross-platform#on-device-llm

FAQ

What is LiteRT-LM?
Google's open-source framework for running large language models on edge devices across Android, iOS, Web, desktop and Raspberry Pi. LiteRT-LM is an orchestration layer for running LLMs on LiteRT, Google's on-device inference runtime. It targets developers who want to run models such as Gemma, Llama, Phi-4 and Qwen locally on phones, browsers, desktops and IoT boards instead of calling a cloud API.
Is LiteRT-LM free to use? What license does it have?
LiteRT-LM is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is LiteRT-LM still maintained?
Actively maintained. The last commit to LiteRT-LM was on Oct 22, 2026.
What can I use LiteRT-LM for?
Building an Android or iOS app that runs a chat model offline. Running a local LLM on a laptop or Raspberry Pi from the command line. Adding on-device function calling to an app without a cloud backend. Serving local embedding models through an OpenAI-style API.
How do I install LiteRT-LM?
Install the LiteRT-LM CLI with uv: uv tool install litert-lm Run the Gemma 3n model from Hugging Face in your terminal: litert-lm run \ --from-huggingface-repo=google/gemma-3n-E2B-it-litert-lm \ gemma-3n-E2B-it-int4 \ --prompt="What is the capital of France?"
Where can I get LiteRT-LM?
The source code and installation instructions are on GitHub: https://github.com/google-ai-edge/LiteRT-LM

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects