Repolode
Submit
lycorp-jp

lycorp-jp/sim-use

A CLI that lets AI agents observe and act on iOS Simulator and Android emulator or device screens through one command set.

View on GitHub
Stars
1,447
Forks
104
Open issues
23
Last commit
7 hours ago

What is it?

sim-use is a cross-platform command-line tool that gives AI agents a way to read and control mobile screens. The `ui` command turns the current screen into a compact text outline of accessibility elements with short aliases such as @9, and `tap @9` acts on that element without needing coordinates. An observe, act, verify loop can run through the same verbs and flags on both iOS Simulator and Android emulators or devices.

On iOS it uses Apple's Accessibility APIs and the Simulator HID pipeline, with a per-device background daemon to reduce per-call startup cost. On Android it talks to an on-device bridge APK over adb forwarding. Other commands cover text entry, paste, gestures, hardware buttons, screenshots, video recording and streaming, app state, and a local web Viewer. Experimental support for physical iPhones and iPads is limited to accessibility-based observation, element taps and screenshots.

The project is written in Swift, is built from source on macOS 14+, and can be installed through Homebrew. It also includes an installable agent skill for AI clients.

Key features

  • 01Renders screens as a token-efficient accessibility outline with @N aliases.
  • 02Taps elements by alias, accessibility id, or label without coordinates.
  • 03Uses one command surface for iOS Simulator and Android emulators or devices.
  • 04Provides JSON output with a shared envelope for scripting and agent loops.
  • 05Records video and GIFs, and streams H.264 from iOS and Android.
  • 06Includes a local web Viewer for inspecting the accessibility tree.

Use cases

  • Letting an AI coding agent build an app, then verify the result on a simulator.
  • Writing automated UI checks that run the same way on iOS and Android.
  • Recording demo GIFs or videos of mobile app flows from the command line.
  • Inspecting which elements an app exposes to accessibility services.

Quick start

  1. 1

    Tap the Homebrew tap

    brew tap lycorp-jp/tap
  2. 2

    Install sim-use with Homebrew

    brew install lycorp-jp/tap/sim-use
  3. 3

    Check that the CLI runs

    sim-use --help

Things to know

  • The iOS Simulator features require macOS, and building from source requires macOS 14+ and Xcode.
  • Building from source needs the large idb XCFrameworks built locally with ./scripts/build.sh dev.
  • Linux support covers only the Android backend; iOS verbs, video capture, and init are macOS-only.
  • Physical iPhone and iPad support is experimental and limited to a subset of verbs.

Topics

#ai-agents#mobile-automation#ios-simulator#android-emulator#accessibility#cli#ai-development

FAQ

What is sim-use?
A CLI that lets AI agents observe and act on iOS Simulator and Android emulator or device screens through one command set. sim-use is a cross-platform command-line tool that gives AI agents a way to read and control mobile screens. The `ui` command turns the current screen into a compact text outline of accessibility elements with short aliases such as @9, and `tap @9` acts on that element without needing coordinates. An observe, act, verify loop can run through the same verbs and flags on both iOS Simulator and Android emulators or devices.
Is sim-use free to use? What license does it have?
sim-use is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is sim-use still maintained?
Actively maintained. The last commit to sim-use was on Oct 21, 2026.
What can I use sim-use for?
Letting an AI coding agent build an app, then verify the result on a simulator. Writing automated UI checks that run the same way on iOS and Android. Recording demo GIFs or videos of mobile app flows from the command line. Inspecting which elements an app exposes to accessibility services.
How do I install sim-use?
Tap the Homebrew tap: brew tap lycorp-jp/tap Install sim-use with Homebrew: brew install lycorp-jp/tap/sim-use Check that the CLI runs: sim-use --help
Where can I get sim-use?
The source code and installation instructions are on GitHub: https://github.com/lycorp-jp/sim-use

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects