Repolode
Submit
vercel-labs

vercel-labs/agent-desktop

A CLI and C library that lets AI agents read and operate desktop apps on macOS, Windows and Linux via OS accessibility trees.

View on GitHubWebsite
Stars
18,740
Forks
1,367
Open issues
26
Last commit
18 hours ago

What is it?

agent-desktop gives AI agents computer use on the desktop without relying on screenshots or pixel matching. A snapshot returns JSON listing every element an agent can act on, each with a stable ref. The agent acts on a ref, takes a new snapshot to verify the result, and repeats, with the decision loop kept in the agent itself.

It runs on macOS, Windows and Linux, and is distributed as an npm package with native binaries for x64 and ARM64, or as release archives. Linux support depends on AT-SPI, and some features differ by platform and display server, such as X11 versus Wayland. Ref actions are headless by default, moving neither the pointer nor focus, and a global --headed flag enables physical input.

The project also ships a C FFI library so programs can call it in process, with Python examples in the README. Snapshots and screenshots may contain private desktop content, so the project documents keeping them out of public logs.

Key features

  • 01Returns accessibility-tree snapshots as JSON with a ref for each actionable element.
  • 02Refs are tied to their snapshot, and stale or ambiguous targets are reported as errors.
  • 03Headless ref actions avoid moving the pointer or taking focus by default.
  • 04Covers observing, clicking, typing, window management, clipboard and notifications.
  • 05Ships a C FFI library for in-process calls from other languages.

Use cases

  • Letting an AI agent operate native desktop apps that have no API.
  • Automating repetitive GUI workflows across macOS, Windows or Linux.
  • Building computer-use agents that verify each action with a fresh snapshot.
  • Embedding desktop automation in other programs through the C library.

Quick start

  1. 1

    Install the package globally with npm

    npm install -g agent-desktop
  2. 2

    Check the installed version

    agent-desktop version
  3. 3

    Grant Accessibility permission on macOS (see the guide)

    agent-desktop permissions --guide
  4. 4

    Take an interactive snapshot of an app such as Finder

    agent-desktop snapshot --app Finder -i

Things to know

  • Accessibility permission is required on macOS, and Linux needs a graphical session with at-spi2-core.
  • Several features are platform-limited, such as Wayland screenshots and GNOME-only notifications on Linux.
  • Key-down, key-up, mouse-down and mouse-up commands are not operational yet.
  • Snapshots and screenshots can expose private desktop content, so keep them out of public logs.

In collections

Open-source AI agent frameworks & appsMCP serversCLI & terminal tools

Topics

#computer-use#accessibility#desktop-automation#ai-agents#cli#rust#mcp#accessibility-api#automation#computer-use-agent

FAQ

What is agent-desktop?
A CLI and C library that lets AI agents read and operate desktop apps on macOS, Windows and Linux via OS accessibility trees. agent-desktop gives AI agents computer use on the desktop without relying on screenshots or pixel matching. A snapshot returns JSON listing every element an agent can act on, each with a stable ref. The agent acts on a ref, takes a new snapshot to verify the result, and repeats, with the decision loop kept in the agent itself.
Is agent-desktop free to use? What license does it have?
agent-desktop is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is agent-desktop still maintained?
Actively maintained. The last commit to agent-desktop was on Nov 7, 2026.
What can I use agent-desktop for?
Letting an AI agent operate native desktop apps that have no API. Automating repetitive GUI workflows across macOS, Windows or Linux. Building computer-use agents that verify each action with a fresh snapshot. Embedding desktop automation in other programs through the C library.
How do I install agent-desktop?
Install the package globally with npm: npm install -g agent-desktop Check the installed version: agent-desktop version Grant Accessibility permission on macOS (see the guide): agent-desktop permissions --guide Take an interactive snapshot of an app such as Finder: agent-desktop snapshot --app Finder -i
Where can I get agent-desktop?
The source code and installation instructions are on GitHub: https://github.com/vercel-labs/agent-desktop

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects