Repolode
Submit
cactus-compute

cactus-compute/cactus

A C++ inference engine and runtime for running LLM, speech and vision models on phones, wearables, smart home devices and robots.

View on GitHubWebsite
Stars
6,148
Forks
519
Open issues
115
Last commit
27 days ago

What is it?

Cactus is a hybrid edge-cloud AI engine aimed at mobile devices and wearables. It provides an OpenAI-compatible API for chat completion, streaming, tool calling, transcription, embeddings, RAG and vision, and it can hand a query off to a cloud model when the local model's confidence falls below a threshold.

The stack has four layers: an engine API, a zero-copy computation graph, ARM NEON and GPU kernels tuned for Apple, Samsung and Pixel hardware, and a custom rotation-based quantization scheme (Cactus Quants) that ranges from 4-bit down to 1-bit. The project publishes benchmarks for Gemma, Qwen, Liquid, Whisper and Parakeet models on Mac and iPhone hardware.

Cactus can be installed on macOS with Homebrew, and it includes a CLI for running, downloading, converting and serving models, plus bindings for Swift, Kotlin, Flutter, React Native, Python and Rust.

Key features

  • 01OpenAI-compatible local HTTP server and C API for chat, streaming and tool calling.
  • 02Speech transcription and vision-language inference on device.
  • 03Automatic cloud handoff based on local model confidence.
  • 04Custom CQ quantization from 4-bit down to 1-bit.
  • 05ARM NEON and GPU kernels for Apple, Samsung and Pixel hardware.
  • 06Bindings for Swift, Kotlin, Flutter, React Native, Python and Rust.

Use cases

  • Running a private on-device assistant in a mobile or desktop app without sending data to a server.
  • Adding offline speech transcription to a wearable or smartphone app.
  • Serving a local OpenAI-compatible endpoint for development on a Mac.
  • Deploying small quantized models on smart home devices or robots.

Quick start

  1. 1

    Install Cactus with Homebrew (macOS)

    brew install cactus-compute/cactus/cactus
  2. 2

    Run a model with the Cactus CLI

    cactus run

Things to know

  • The license is not clearly specified (metadata reports NOASSERTION), so check the repository before commercial use.
  • The quick demo is shown for Mac, and the benchmark tables focus on Apple devices.
  • Converting arbitrary HuggingFace models is described as experimental; only some model families are especially tested.

In collections

Run LLMs locallyAI video & audio toolsAndroid apps & phone tools

Topics

#edge-ai#on-device-ai#llm-inference#quantization#mobile-inference#speech-recognition#rag#arm-neon#ai#android

FAQ

What is cactus?
A C++ inference engine and runtime for running LLM, speech and vision models on phones, wearables, smart home devices and robots. Cactus is a hybrid edge-cloud AI engine aimed at mobile devices and wearables. It provides an OpenAI-compatible API for chat completion, streaming, tool calling, transcription, embeddings, RAG and vision, and it can hand a query off to a cloud model when the local model's confidence falls below a threshold.
Is cactus free to use? What license does it have?
cactus is open source under the Other license. Check the license terms before using it in commercial products.
Is cactus still maintained?
Actively maintained. The last commit to cactus was on Sep 26, 2026.
What can I use cactus for?
Running a private on-device assistant in a mobile or desktop app without sending data to a server. Adding offline speech transcription to a wearable or smartphone app. Serving a local OpenAI-compatible endpoint for development on a Mac. Deploying small quantized models on smart home devices or robots.
How do I install cactus?
Install Cactus with Homebrew (macOS): brew install cactus-compute/cactus/cactus Run a model with the Cactus CLI: cactus run
Where can I get cactus?
The source code and installation instructions are on GitHub: https://github.com/cactus-compute/cactus

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects