Do you want to get an AI agent to do something to an iPhone. You could try and jailbreak the device and do some heavy software hacking, or… you could …. [Henry Yuan] explains how to achieve this with an ESP32.
The first step is getting an AI agent access to the current state of the phone screen—which [Henry] achieves with AirPlay screen mirroring and UxPlay. A Gstreamer pipeline is then used to feed the screen as a JPEG stream to the AI agent of your choice. Then comes control, which is achieved by an ESP32-S3 acting as a USB-HID device, emulating both USB mouse and keyboard.
With iOS AssistiveTouch enabled, this gives the agent control over inputs to the device. From there, it’s up to the AI agent to figure out the actions to make to achieve the desired behavior, which is obviously very use-case specific.
Discover more from ChuckysCarnage
Subscribe to get the latest posts sent to your email.
