Repolode
Submit
onehumanbeing

onehumanbeing/esp32-iphone-agent

Lets an AI vision model operate a real iPhone using a $7 ESP32-S3 board that acts as an AirPlay screen receiver and USB HID mouse, with no computer needed at run time.

View on GitHub
Stars
3,302
Forks
440
Open issues
14
Last commit
11 hours ago

What is it?

ESP32 iPhone Agent is a system that lets an AI model control a physical iPhone. The ESP32-S3 board works as an AirPlay receiver, so the phone's screen is mirrored to it over Wi-Fi, and it also appears to the phone as a standard USB mouse and keyboard. An iPhone app on the same device decodes the mirrored stream, sends screenshots to an OpenAI-compatible vision model, and carries out taps, swipes, typing and key presses based on the model's replies.

The project targets setups where no Mac, developer account or jailbreak is available. Flashing the board and building the app require a Mac with Xcode once, but running the agent does not. Each run is saved on the phone so it can be replayed step by step, and traces can be exported for analysis by another agent.

The project is MIT-licensed, except for a firmware component ported from UxPlay that is GPL-3.0. Requirements include iOS 18 or later and cellular data, because the phone's Wi-Fi is occupied by the board's hotspot while the agent runs.

Key features

  • 01Receives the iPhone screen over AirPlay Screen Mirroring, with H.264 passed through to the app over BLE.
  • 02Acts as a USB HID mouse and keyboard, using AssistiveTouch for pointer-based taps and swipes.
  • 03Works with any OpenAI-compatible vision model, including self-hosted servers such as vLLM and Ollama.
  • 04Runs planner and actor steps in the background and pauses or resumes when the board or mirroring link drops.
  • 05Saves every run for step-by-step replay and export of screenshots and actions.

Use cases

  • Prototyping an AI agent that operates iOS apps without a Mac or developer account.
  • Automating repetitive tasks on an iPhone you own, such as setting alarms or changing settings.
  • Researching computer-use agents on a physical device rather than a simulator.
  • Collecting run traces of phone interactions for analysis by another agent.

Topics

#ai-agent#esp32#iphone#usb-hid#airplay#computer-use#llm#automation#esp32-s3

FAQ

What is esp32-iphone-agent?
Lets an AI vision model operate a real iPhone using a $7 ESP32-S3 board that acts as an AirPlay screen receiver and USB HID mouse, with no computer needed at run time. ESP32 iPhone Agent is a system that lets an AI model control a physical iPhone. The ESP32-S3 board works as an AirPlay receiver, so the phone's screen is mirrored to it over Wi-Fi, and it also appears to the phone as a standard USB mouse and keyboard. An iPhone app on the same device decodes the mirrored stream, sends screenshots to an OpenAI-compatible vision model, and carries out taps, swipes, typing and key presses based on the model's replies.
Is esp32-iphone-agent free to use? What license does it have?
esp32-iphone-agent is open source under the MIT license. Check the license terms before using it in commercial products.
Is esp32-iphone-agent still maintained?
Actively maintained. The last commit to esp32-iphone-agent was on Oct 19, 2026.
What can I use esp32-iphone-agent for?
Prototyping an AI agent that operates iOS apps without a Mac or developer account. Automating repetitive tasks on an iPhone you own, such as setting alarms or changing settings. Researching computer-use agents on a physical device rather than a simulator. Collecting run traces of phone interactions for analysis by another agent.
Where can I get esp32-iphone-agent?
The source code and installation instructions are on GitHub: https://github.com/onehumanbeing/esp32-iphone-agent

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects