Repolode
Submit
simular-ai

simular-ai/Agent-S

An open-source computer use agent framework that controls a real GUI with mouse, keyboard and screen to complete tasks in desktop and web apps.

View on GitHubWebsite
Stars
12,660
Forks
1,707
Open issues
64
Last commit
12 days ago

What is it?

Agent S is an open-source framework for building computer use agents. Given a natural-language task, it looks at the screen and completes the work by clicking, typing and scrolling in ordinary desktop and web applications, without requiring API integrations or per-app scripts.

The current generation, Agent S3, reports 72.60% on the OSWorld benchmark. The project runs on macOS, Windows and Linux, and can use models from OpenAI, Anthropic, Gemini, OpenRouter and vLLM, paired with a separate grounding model such as UI-TARS for locating screen elements. It is installable with pip as the gui-agents package and can be run from a CLI or a Python SDK.

The framework includes an optional local coding environment that lets the agent execute Python and Bash code for data processing and file tasks. This feature runs arbitrary code with the user's permissions, so the project warns to use it only in trusted settings.

Key features

  • 01Operates desktop and web apps through mouse, keyboard and screen input, with no per-app integration.
  • 02Supports macOS, Windows and Linux.
  • 03Works with OpenAI, Anthropic, Gemini, OpenRouter and vLLM models.
  • 04Uses a separate grounding model (e.g. UI-TARS) to locate screen elements.
  • 05Provides a CLI and a Python SDK, installable via pip as gui-agents.
  • 06Includes an optional local environment for running Python and Bash code.

Use cases

  • Automating repetitive tasks across desktop and web applications.
  • Researching OS-level computer use agents and benchmarking them on OSWorld.
  • Building custom agents that drive GUI software without APIs.

Quick start

  1. 1

    Install the gui-agents package with pip

    pip install gui-agents
  2. 2

    Install tesseract (required by pytesseract on macOS)

    brew install tesseract
  3. 3

    Set your model API key as an environment variable

    export OPENAI_API_KEY=<YOUR_API_KEY>
  4. 4

    Run Agent S3 from the CLI with a grounding model endpoint configured

    agent_s --provider openai --model gpt-5-2025-08-07 --ground_provider huggingface --ground_url http://localhost:8080 --ground_model ui-tars-1.5-7b --grounding_width 1920 --grounding_height 1080

Things to know

  • A grounding model such as UI-TARS-1.5-7B is required, which needs a separate hosted endpoint.
  • The agent controls your real mouse and keyboard and is designed for single-monitor setups.
  • The optional local coding environment runs arbitrary code on your machine, so use it with care.
  • Running it depends on paid model API keys from providers such as OpenAI or Anthropic.

Topics

#computer-use#ai-agents#gui-automation#multimodal-llm#osworld#python#agent-computer-interface#computer-automation#computer-use-agent#cua

FAQ

What is Agent-S?
An open-source computer use agent framework that controls a real GUI with mouse, keyboard and screen to complete tasks in desktop and web apps. Agent S is an open-source framework for building computer use agents. Given a natural-language task, it looks at the screen and completes the work by clicking, typing and scrolling in ordinary desktop and web applications, without requiring API integrations or per-app scripts.
Is Agent-S free to use? What license does it have?
Agent-S is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is Agent-S still maintained?
Actively maintained. The last commit to Agent-S was on Oct 8, 2026.
What can I use Agent-S for?
Automating repetitive tasks across desktop and web applications. Researching OS-level computer use agents and benchmarking them on OSWorld. Building custom agents that drive GUI software without APIs.
How do I install Agent-S?
Install the gui-agents package with pip: pip install gui-agents Install tesseract (required by pytesseract on macOS): brew install tesseract Set your model API key as an environment variable: export OPENAI_API_KEY=<YOUR_API_KEY> Run Agent S3 from the CLI with a grounding model endpoint configured: agent_s --provider openai --model gpt-5-2025-08-07 --ground_provider huggingface --ground_url http://localhost:8080 --ground_model ui-tars-1.5-7b --grounding_width 1920 --grounding_height 1080
Where can I get Agent-S?
The source code and installation instructions are on GitHub: https://github.com/simular-ai/Agent-S

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects