How to Use iPhone-Use: Control a Real iPhone With an AI Agent (USB Setup, First Tap, and the MCP Arc)

CodexFuse Team · 2026-11-06T21:26:07.679Z · 22 min read min read

What Is iPhone-Use, and What Does It Actually Do?

Learning how to use iphone-use means learning to let an AI agent operate a real iPhone over USB, in plain language, while you watch its screen on your Mac. iPhone-Use is the open-source tool from zhongerxin on GitHub that connects Codex to a physical iPhone through WebDriverAgent, so the agent can open apps, read screens, tap, scroll, and type. The hyphenated name matters, because iphone-use is a software project, not a guide to using an iPhone, and this iphone-use tutorial is about the tool only.

As of November 2026, this guide describes iPhone-Use v0.3.16, the current release, published on 2026-10-22, and every step below was checked against it. The project is 32 days old, it holds 4,928 stars and 491 forks on GitHub, which works out to 10.04 stars per fork and a 9.96 percent fork ratio, and its last code push was 16 days before this guide. It is free open source under the MIT license, with 33 open issues, or 0.67 open issues per 100 stars, and it gains about 154.0 stars per day.

CodexFuse offers no iPhone-Use tool, and this guide teaches the open-source software itself. You will run it on your own Mac, only with an iPhone you own, and only with apps and accounts that are yours. If you would rather drive a browser than a phone, the Browser-Use beginner guide covers the sibling tool that does the same job for Chrome, and this guide covers only the phone side.

Think of it as a translator between two worlds: the agent speaks in words, and WebDriverAgent turns those words into the same taps a finger would make. To follow the later sections, it helps to know what an API is, because iPhone-Use adds that kind of contract, called MCP, between Codex and your phone. The rest of this guide walks the full arc, from install to first tap to the MCP setup, with the honest limits at the end.

What Do You Need Before You Install iPhone-Use?

You need a Mac, full Xcode, a free or paid Apple account, a real iPhone, and a USB cable, because iPhone-Use builds and signs a helper app on your phone and cannot run on Windows or Linux. The Mac runs the local MCP server and the live screen sidebar, while the iPhone runs WebDriverAgent, the Apple test runner that receives the taps. Nothing here needs a jailbreak, and nothing runs in the cloud: the whole system stays on your desk.

The single most common failure is installing only the Xcode Command Line Tools, which are not enough. You need the full Xcode app from the Mac App Store, opened once so its first-launch setup finishes, and it must support your iPhone's iOS version. A free Apple ID works, with one honest catch that gets its own section later: its signing profile expires after 7 days.

Check the versions before you start, because the Node floor is oddly specific: Node 20.19 or newer, 22.12 or newer, or 24 or newer, with npm 10 or newer. Python 3.9 or newer runs the MCP server on the standard library alone, so no Python packages need installing. The iphone-use github page lists every requirement in its dependency table, so this section follows that table row by row.

How Do You Install iPhone-Use?

There is no pip install and no npm install, because iPhone-Use installs as a Codex plugin from a git clone plus one shell script. The script stages the source, adds the plugin to Codex, and registers the MCP server named iphone_use, and then you reconnect the Codex chat to load the tools. The whole iphone-use install is one clone, one script, and one reconnect.

1. Install full Xcode from the Mac App Store, open it once, and let its first-launch setup finish.
2. Sign in to Xcode with your Apple ID and confirm your development team is usable.
3. Connect the iPhone by USB, tap Trust on the phone, and turn on Developer Mode if asked.
4. Clone the repository into a local directory you can find again.
5. Run the install script from inside the cloned directory.
6. Reconnect the Codex chat so the iPhone Use tools and skills load.
7. Ask Codex to run the setup skill, verify READY, and show the phone screen.

git clone https://github.com/zhongerxin/iPhone-use.git
cd iPhone-use
sh scripts/install.sh

Ask Codex: use iphone-use-setup to configure the USB iPhone, install and start WDA, verify READY, then show the phone screen.

If git is new to you, the Git beginner guide covers clone and pull before you start. The install script needs the Codex CLI, so the Codex beginner guide is the place to start if Codex is not installed yet. Run the script once, and update later with git pull --ff-only followed by the same script, then reconnect the chat.

git pull --ff-only
sh scripts/install.sh

The script does three things in order: it stages the source, adds the plugin marketplace entry, and pipes the result into the MCP registration step. You do not type any of that by hand, and you should never reuse the author's device identifiers or signing configuration, which the README warns against. The README also says the repository is currently private, but that line is stale: the repo is public and cloneable, and both independent reviews from October flagged the mistake.

How Do You Set Up WebDriverAgent and Sign It?

WebDriverAgent, usually called WDA, is the Apple test runner that lives on your iPhone and receives the agent's taps, and the iphone-use webdriveragent setup pins it to a verified 16.14.0 commit. The setup skill downloads that pinned WDA, configures signing with your own Apple development team and a signable bundle ID, builds it, and starts it on the phone. You never sign with anyone else's team, and you never paste anyone else's bundle ID.

The recommended route is agent-driven: you paste the install prompt into Codex, and the iphone-use-setup skill discovers the device, handles the signing, and starts WDA for you. The manual route exists for people who want to see the machinery, and the table below compares the two. Either way, the WDA service must stay running while the agent works, and the default forwarded port is 18100 on your local loopback, with remote addresses rejected.

If the build fails, the usual cause is a signing problem: the wrong team selected, a bundle ID that is already taken, or Xcode still finishing its first launch. The troubleshooting doc in the repo lists the profile-expired case first, because a free profile stops working after 7 days and the fix is a rebuild, not a restart. Rebuild with the same local configuration, then check READY again.

How Do You Connect Your iPhone for the First Time?

The first connection is a readiness check, not a task: you ask Codex to run the setup skill, and you start phone tasks only when pua_ready returns ready=true. The setup flow first calls pua_setup with the status action, which reuses a healthy service instead of starting a second one, and only starts a fresh install when nothing is running. Then one call to pua_screen shows the live phone screen in the sidebar.

pua_setup(action="status")
pua_ready
pua_screen()

The canonical first verification is the Calculator test from the README: open the iPhone Calculator and verify 12×34=408. That one check proves the whole chain at once, because the agent must read the screen, find the keys, tap them in order, and read the result back. If the result reads 408, the phone, the signing, the USB forward, and the agent are all working.

Keep the phone unlocked and the cable connected for this first run, and keep the Mac awake. If the screen preview stays blank, check that WDA is still running on the phone and that the phone still trusts the Mac. A locked phone or an untrusted cable is the most common reason a first connection stalls.

What Can the Agent Actually Do on Your iPhone?

The agent has 18 model-accessible tools plus 2 widget-only tools, all named with the pua_ prefix, which stands for Phone Use Agent. Older guides and videos may show wda_ names, which belonged to versions up to 0.3.5, so a command that looks familiar but fails is usually a stale name. The tools split into four jobs: check the setup, see the screen, act on it, and collect results.

Two rules govern everything the agent does: passwords, verification codes, and Face ID are handed to you, never done by the agent, and typing and submitting are separate steps, because the tool does not submit text by default. On element-targeting failure, the agent inspects a screenshot and tries a coordinate tap, which is the fallback the README documents. The screenshot tool, pua_screenshot, arrived in v0.3.15 and saves a system screenshot to the phone's Photos.

The live screen widget is the part you will actually watch: it shows the phone in the Codex sidebar while the agent works. The state lives by default in ~/.local/share/iphone-use/, with run directories at mode 700 and private config and image files at mode 600. Signing material, device data, logs, and dependency directories are excluded from source packages.

How Do You Give the Agent Its First Real Task?

A first real task should be small, visible, and reversible: ask the agent to open Notes, type a short sentence, and read it back to you. That task exercises the full loop, which is launch, observe, act, and verify, without touching any account or payment. If any step fails, the agent sees the same screen you see and can say so in plain words.

Open Notes on my iPhone, type "First tap with iPhone-Use", then read the note back to me.

Good tasks name the app, the action, and the check: open the app by name, do one thing, and report what changed. Vague tasks like tidy my phone invite the agent to guess, and guessing on a real phone is how settings change by accident. Start with one app at a time, and keep the phone where you can see it while the agent works.

Banking and health apps are the showcase case, because they have no API and often block screen capture, yet the agent reads them through the accessibility tree instead of pixels. That is the ai agent control iphone idea in one line: the agent drives the real build on a real device, which is why the tool is useful for testing your own app under development. Never point it at someone else's phone or accounts, and only use it on a phone you own.

How Does the iPhone-Use MCP Server Work?

The MCP server is the bridge that lets Codex, or any MCP client, send tool calls to the phone, and the install script registers it as the standard MCP server named iphone_use. MCP is the Model Context Protocol, and to follow this section it helps to know what MCP is: a shared way for an AI program to call tools outside itself. The server runs over stdio, which means Codex starts it as a local process and talks to it through standard input and output.

{
  "mcpServers": {
    "iphone_use": {
      "type": "stdio",
      "command": "python3",
      "args": ["${PLUGIN_ROOT}/server/iphone_use.py"],
      "cwd": "${PLUGIN_ROOT}"
    }
  }
}

Codex needs no manual config, because the install script registers the server for you. A generic MCP client can use the same stdio command with the checkout path in place of the PLUGIN_ROOT placeholder, which is python3 followed by the path to server/iphone_use.py. There is no official Claude Code support in this repo, so do not expect a Claude plugin: the project is Codex-first, and any Claude Code setup is a generic MCP connection, not an official one.

The server keeps one daemon per phone on the Mac, reads the screen as an accessibility tree of text rather than pixels, and checks that the screen has settled before it answers. Local state stays on your Mac, the WDA forward stays on local loopback port 18100, and remote addresses are rejected. That local-only shape is the privacy story: your screen never leaves the desk.

How Do You Use iPhone-Use Without Codex?

Codex is the primary host, but the repo ships two adapters that need no Codex CLI: WorkBuddy and Doubao Work. Both hosts control a real iPhone over USB, show the live screen in a sidebar, and share the same local device configuration and WDA builds, so a setup you already did is reused. Their analytics are off by default, unlike the Codex route.

python3 workbuddy/install.py
python3 doubao/install.py

For WorkBuddy, you run the installer from the source directory, then trust and enable iphone_use in Connector Management and start a new chat. For Doubao Work, you run its installer, create an iphone_use stdio connector from the installer output, and upload the iphone-use and iphone-use-setup skill folders. The setup prompt and the Calculator check are the same on all three hosts.

If you use the Rust project by the same name, stop and check the repository first, because three different projects share the name iphone-use, and their install steps are not interchangeable. This guide covers zhongerxin/iPhone-use only, the Python Codex plugin with 4,928 stars, which is 10.62 times the 464 stars of the Rust project. The leeguooooo project is a Rust rewrite with its own docs site, and the hughlee824 project is a Swift command-line tool.

iPhone-Use vs Browser-Use: Which Agent Controls What?

The iphone-use vs browser-use comparison is really a question about surfaces: browser-use drives a real Chromium browser through the Chrome DevTools Protocol, while iphone-use drives a real iPhone through WebDriverAgent. Both are open source, both are Python, both take plain-language tasks, and both expose their work as tools an agent can call. The difference is the hardware gate: browser-use needs only Python and Chrome, while iphone-use needs a Mac, full Xcode, a signed runner, and a USB iPhone.

The numbers show the age gap plainly: browser-use holds 120,207 stars against iphone-use at 4,928, which is 24.39 times more, a gap of 115,279 stars, and browser-use is about two years old while iphone-use is 32 days old. Browser-use also works out to 8.96 stars per fork, close to iphone-use at 10.04, which says both projects attract a similar share of contributors. Age explains the gap, not quality: browser-use has had two years to grow, and iphone-use gained its 4,928 stars in 32 days.

Choose browser-use when the job lives in a web app, and iphone-use when the job lives in a phone app with no API. Mobile-use is the closest direct rival, a cross-platform agent framework for Android and iOS that claims a perfect AndroidWorld benchmark score, while Appium is the test framework underneath it all, not a rival at all. Maestro covers iOS simulators only, so the real-iPhone job is exactly the gap iphone-use fills.

What Are the Limits and Honest Trade-Offs?

The honest limit is the 7-day signing rule: a free Apple ID issues provisioning profiles that expire 7 days from issuance, so the WDA Runner stops launching on day eight and you rebuild and re-sign with the same local configuration. That is 52.1 re-signs a year if you keep using it, against a paid Apple Developer Program membership at $99 a year, which extends profiles to up to a year. A free account can also register up to 10 App IDs and 3 devices, with up to 3 apps per device.

The release cadence is the second honest signal: 34 releases landed in 16 days, which is 2.12 releases a day, and then no commits for 16 days, with the latest release v0.3.16 published on 2026-10-22. The repo is not archived, and its metadata was updated on 2026-11-06, but this guide does not claim daily active development. Pin your expectations to v0.3.16, and check the releases page before you install.

Analytics are on by default on the Codex route: anonymous usage statistics go to the project's PostHog project and GA4 property, including tool calls and launched app identifiers, but never phone images, input text, or device identifiers. Disable them with IPHONE_USE_ANALYTICS=0 or DO_NOT_TRACK=1 and restart the MCP server. The Mac-only shape, the Xcode requirement, and the USB cable are permanent limits, not bugs.

IPHONE_USE_ANALYTICS=0
DO_NOT_TRACK=1
# restart the MCP server after setting either variable

What Are the Seven Mistakes Beginners Make?

Most first installs fail for the same seven reasons, and every one of them is cheap to avoid once you know it. The table below is the short version, and the sections above carry the detail. Read it before you clone, not after the build fails.

If you hit a failure that is not in the table, run pua_doctor first, because it checks the environment and names the missing piece before you guess. The repo's troubleshooting doc lists the expired-profile case first, which is the one beginners meet most. When in doubt, rebuild rather than restart: a stale build explains more failures than a stale connection.

from datetime import date
today = date(2026, 11, 7)
repo_age = (today - date(2026, 10, 6)).days
rel_age = (today - date(2026, 10, 22)).days
stars_per_fork = round(4928 / 491, 2)
fork_ratio = round(491 / 4928 * 100, 2)
issues_per_100 = round(33 / 4928 * 100, 2)
stars_per_day = round(4928 / repo_age, 2)
browser_ratio = round(120207 / 4928, 2)
star_gap = 120207 - 4928
leeguooooo_age = (today - date(2026, 6, 9)).days
zhongerxin_vs_leeguooooo = round(4928 / 464, 2)
releases_per_day = round(34 / 16, 2)
resigns_per_year = round(365 / 7, 1)
print(repo_age, rel_age, stars_per_fork, fork_ratio, issues_per_100, stars_per_day, browser_ratio, star_gap, leeguooooo_age, zhongerxin_vs_leeguooooo, releases_per_day, resigns_per_year)
# Output: 32 16 10.04 9.96 0.67 154.0 24.39 115279 151 10.62 2.12 52.1

Sources

The project facts in this guide come from the iphone-use github repository by zhongerxin, fetched on 2026-11-07: the repository API record, the releases list, the latest release v0.3.16, the English README, the install script, the MCP configuration files, and the setup troubleshooting doc. The same day's API reads covered the leeguooooo/iphone-use project for the name-collision check and browser-use/browser-use for the comparison. Apple's own membership comparison page confirmed the 7-day free provisioning rule, and the Appium XCUITest device setup guide, linked by the README, confirmed the signing requirements.

Independent coverage checked the same day includes the Frontier Now review from 2026-10-13, the Hysenlabs project analysis updated three days ago, the Agent Daily deep dive from 13 days ago, the Kaelzhang teardown from 27 days ago, and the aiidelist step-by-step guide, which cites v0.3.7 and is now stale. A YouTube overview from 15 days ago walks the prerequisites and the Calculator check, and a dev.to listicle from October 2026 gives a wrong install command, which this guide does not repeat. The Expo blog on the Apple Developer Program and the BetaDrop free-account guide corroborated the 7-day expiry and the paid-program alternative.

Comparison sources include the Jacar browser-use guide from 2026-09-02 and the Andrew Ooo browser-use automation post from October 2026 for browser-use facts, the Medium Maestro versus Appium comparison for the simulator-only limit, the Momentic mobile MCP setup post for the Appium and Maestro framing, and the Brightcoding mobile-use post for the cross-platform rival. Star and fork counts are a 2026-11-07 snapshot, and no testbed outputs were executed for this guide, so every command above is quoted from the project's own README and scripts, not from a local run. The Rust namesake is 151 days old with 464 stars, against 32 days and 4,928 stars for the project this guide covers.

Frequently Asked Questions

Is iPhone-Use free to use?

Yes, iPhone-Use is free and open source under the MIT license, with no paid tier and no license key. The only optional cost is the Apple Developer Program at $99 a year, which you need only if you want signing profiles that last up to a year instead of 7 days. Codex itself is your own subscription, and it is not part of this tool.

Does iPhone-Use work on Windows or with an Android phone?

No, iPhone-Use needs macOS, full Xcode, and a real iPhone connected by USB, so Windows and Android are out. Xcode must support your iPhone's iOS version, which is the only version rule the README states. If your job lives in a browser instead, browser-use is the Windows-friendly sibling.

How do I know the setup actually worked?

Run the readiness check and wait for pua_ready to return ready=true, then call pua_screen once to see the live phone screen. The README's own test is to open the iPhone Calculator and verify 12×34=408, which proves the screen read, the taps, and the result read in one check. If the result reads 408, the whole chain works.

Why does iPhone-Use stop working after 7 days?

A free Apple ID issues provisioning profiles that last only 7 days, so the WDA Runner stops launching on day eight. Rebuild and re-sign with the same local configuration, then check READY again, and the phone works for another 7 days. A paid Apple Developer Program membership extends profiles to up to a year.

Can I use iPhone-Use with Claude Code instead of Codex?

Not officially, because this repo is Codex-first and ships no Claude Code plugin or config. A generic MCP client can start the same stdio server with python3 and the path to server/iphone_use.py, but that is an inferred connection, not a documented one. If you want documented Claude Code registration, you are looking at the other iphone-use project, the Rust one by leeguooooo, which is a different tool.