What Is Vercel's agent-desktop? The Beginner's Guide to Agent Computer Use Without Screenshots
CodexFuse Team · 2026-11-04T19:49:29.579Z · 12 min read min read
What Is Vercel's agent-desktop? The Beginner's Guide to Agent Computer Use Without Screenshots
If you have been wondering what is agent desktop and why it topped GitHub's daily trending list on November 3, 2026, here is the short version: it is Vercel Labs' open-source Rust CLI that gives AI coding agents computer use on real desktop apps without screenshots or pixel matching. Instead of photographing your screen and asking a vision model to guess where a button is, agent-desktop reads the operating system's accessibility tree, hands the agent a JSON snapshot where every clickable element has a stable ref, and the agent clicks or types on those refs. It works with the agent you already use — Claude Code, Codex, Cursor's CLI, or a plain shell script — because the reasoning loop lives in your agent, not in the tool.
What is agent-desktop, exactly?
agent-desktop is computer-use infrastructure for AI agents, not an AI agent itself. The project's own tagline says it plainly: "Computer use for AI agents: see and operate desktop apps on macOS, Windows, and Linux through the OS accessibility tree." Every command prints one JSON envelope and exits; there is no model inside the tool. The observe-act-verify loop runs in the calling agent, which is why any agent that can run shell commands can drive it.
The README summary puts the whole idea in one paragraph: "agent-desktop gives an AI agent computer use without screenshots or pixel matching. It reads the accessibility tree of a desktop app and returns it as JSON, with a ref for every element the agent can act on. The agent clicks or types on a ref, reads the UI again to check the result, and repeats. The loop lives in your agent; agent-desktop has no model inside. It ships as a CLI and a C library." Everything in this agent desktop Vercel explained guide comes from the official README, the docs, and launch-week explainers — every quote is verbatim. As of November 4, 2026, the repository is vercel-labs/agent-desktop, licensed Apache-2.0, with about 14,700 stars after a launch week that started from the personal project of its original author in February 2026.
How does agent-desktop see the screen without screenshots?
agent-desktop asks the operating system what is on screen instead of looking at pixels. Every major OS maintains a structured description of each window for screen readers: roles such as button, treeitem, or window, plus names, values, and the actions each element supports. agent-desktop treats that tree as the source of truth for observation and semantic interaction, and it speaks each platform's native API: the Accessibility API on macOS, UI Automation on Windows, and AT-SPI on Linux.
A snapshot is the key primitive. An agent desktop snapshot is one JSON envelope: a snapshot ID, a count of refs, and a tree of nodes. The agent runs agent-desktop snapshot --app Finder -i and each element the agent can act on gets a ref such as @s8f3k2p9:e8, which works like a handle rather than a coordinate. That is the reliability argument for this whole approach to agentic AI on the desktop: refs do not drift when the window moves, the theme changes, or the display scales, because nothing is inferred from pixels. At its core, agent desktop accessibility tree reading replaces the screenshot pipeline with the structure the OS already maintains.
The ref system also fails safely. Refs belong to the snapshot that produced them, and a bare @e8 without its snapshot prefix is rejected with INVALID_ARGS. If the UI shifts between the snapshot and the action, the agent does not click the wrong thing — it gets a machine-readable STALE_REF or AMBIGUOUS_TARGET error, takes a new snapshot, and retries with fresh refs.
What is the snapshot → act → snapshot loop?
The core loop is three steps: observe, act on a ref, observe again to verify. The README's own quick-start shows the whole pattern in three lines:
agent-desktop snapshot --app Finder -i # interactive elements, each with a ref agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot agent-desktop snapshot --app Finder -i # observe again to verify
The third step is the one beginners skip and should not. The project docs put it bluntly: "A successful action does not prove the UI changed." A click can return success while the app ignored it, a dialog can open somewhere unexpected, or a list can re-render and shuffle every ref. So the loop is deliberately boring — snapshot, act, snapshot again — and the agent is expected to confirm the interface changed the way it intended before moving on.
This loop is also why dense apps get a special pattern. A full accessibility tree of an Electron app like Slack is enormous, so the docs suggest mapping it shallowly first with snapshot --skeleton and then expanding only the region you care about with snapshot --root <ref>. You pay tokens for the part of the UI you are working in, not the whole window.
How do you install agent-desktop and run your first command?
You can have agent-desktop installed and driving an app in about five minutes. This agent desktop tutorial assumes a machine you control and an agent that can run shell commands. Here is the first-run arc:
- Install the CLI. The README's install route is
npm install -g agent-desktop, which ships native x64 and ARM64 binaries; macOS and Linux users can also usebrew install agent-desktop. The whole thing runs fine on a headless Linux box too, including the kind of VPS hosting developers already use for sandboxes. - Check your platform. Run
agent-desktop statusto confirm your OS, architecture, and which surfaces are supported. - Grant the permission. On macOS, grant the app Accessibility permission;
agent-desktop permissions --guidewalks you through it. Windows needs no grant at the terminal's integrity level, and Linux needs the at-spi2-core package. - Take your first snapshot. Run
agent-desktop snapshot --app Finder -iand read the JSON: every interactive element, each with its ref. - Teach your agent the loop. Run
agent-desktop skills get desktopfor the loop, refs, verification, and retry rules, and install them into your harness withnpx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop.
npm install -g agent-desktop agent-desktop status agent-desktop permissions --guide agent-desktop snapshot --app Finder -i npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop
Version note, as of November 4, 2026: v1.0.0 landed October 27–28, 2026 (sources conflict on the exact day), v1.0.2 followed on October 30 with the official docs site, and v1.1.0 is the current release from November 3, bringing Windows fixes and an on-screen overlay that shows where the agent is clicking and typing.
What can agent-desktop actually do? The 60 commands
agent-desktop exposes 60 command names, 56 of them operational. The README is upfront about the other four: key-down, key-up, mouse-down, and mouse-up refuse to run until a daemon exists that can release a held key or button. The operational set falls into nine groups:
| Group | What it covers | Example |
|---|---|---|
| Observe | Snapshot the UI, drill into regions | snapshot --app Slack -i |
| Act on a ref | Click, type, press, scroll elements | click @s8f3k2p9:e8 |
| Keyboard and mouse | Low-level input when refs are not enough | press Enter |
| Apps and windows | Launch, focus, list, manage windows | open, window listing |
| Notifications | Read and manage system notifications | notification commands |
| Clipboard | Read and write the clipboard | clipboard commands |
| Wait | Pause for UI changes before verifying | wait commands |
| Sessions | Keep agent sessions across runs | session commands |
| System | Status, permissions, skills, MCP server | status, mcp |
One design choice runs through all of them: actions are headless by default, meaning no pointer movement and no focus change. The README states it directly: "Actions are headless by default: no pointer movement, no focus change. Pass --headed for physical input." Your agent can click through a settings window while you keep typing in another app.
Which agents can use agent-desktop?
Any agent that can run shell commands. The design is harness-agnostic on purpose: Claude Code, Codex, and Cursor's CLI all qualify, and one tutorial put that to the test by having each of them drive a native calculator app end to end. The agent desktop Claude Code combination has the most public testing: on Debian 13 with agent-desktop 1.0.2, Claude Code completed the calculator task in about 28.6 seconds and Codex in about 35 seconds — real, timed evidence that the loop works across harnesses.
There are two ways to wire it into your agent. The skill route installs the loop, ref, verification, and retry rules as agent skills your harness reads automatically. The MCP route runs agent-desktop mcp, which the README describes in one line: "`agent-desktop mcp` serves every command as an MCP tool over stdio." That gives you tools like desktop_snapshot and desktop_click in any MCP-capable client, no shell required.
Platform coverage is honest and specific. The README's platform table reads: macOS 13 or later on Intel or Apple Silicon via the AX API; Windows 10 1809 or later on x64 or ARM64 via UI Automation; Linux on Ubuntu 24.04 or later, x86_64 or aarch64, X11 or Wayland, via AT-SPI. macOS is the one that asks for the Accessibility grant; Linux is the one where a sandboxed agent may find the desktop D-Bus blocked.
agent-browser vs agent-desktop: what's the difference?
The agent browser vs agent desktop question has a one-line answer: they are sibling tools from Vercel Labs with one clean dividing line — agent-browser drives web pages, agent-desktop drives native desktop apps. When Chris Tate announced agent-desktop on October 30–31, 2026, he billed it as "agent-browser, for your whole computer" — the same token-efficient, ref-based computer use, extended from the browser tab to the entire desktop operating system.
The boundary matters because the tools do not overlap. agent-desktop explicitly does not automate web pages; that is what agent-browser is for. Conversely, agent-browser cannot reach the native settings window, the file manager, or the desktop IDE where your agent might need to click next. Pick the tool by the surface your agent has to touch, and expect future agent workflows to chain both: browser for the web app, desktop for the native one.
What are the limits of agent-desktop?
agent-desktop only works with apps that expose proper accessibility information. Content that is fully custom-drawn, with no accessibility nodes — some canvas-heavy apps and games — is invisible to it, and those cases still need screenshots or other methods alongside it. It also does not automate web pages, which belongs to the sibling browser tool by design.
Treat snapshots as sensitive. The README's privacy warning is explicit: "Snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content. Keep them out of public logs and uploads." A snapshot of your desktop is your desktop, with its open tabs, file names, and clipboard — do not paste it into a public issue or a shared log.
Finally, mind the version line. agent-desktop is young — v1.1.0 is the latest release as of November 4, 2026, and the changelog shows the project is still fixing platform gaps release by release. Expect rough edges on Windows and Linux, check the release notes before you build on a new command, and never treat a builder's performance numbers as benchmarks until the project publishes its own.
Sources
- Vercel Labs — agent-desktop README and releases (vercel-labs/agent-desktop), fetched November 4, 2026
- Official docs — agent-browser.dev/desktop
- explainx.ai — "agent-desktop: Computer Use CLI With 60 Commands (Rust)", November 2026
- aicodewith.ai — "agent-desktop Tutorial: Desktop Computer Use in Claude Code, Codex, Cursor", November 2026
- techtoheart.com — "Vercel's agent-desktop: Native App Control for Shell Agents Without a Vision Model", November 2026
- Medium (Abhinav Dobhal) — "Vercel agent-desktop: Rust Computer Use for AI Agents", November 2026
- dev.to (piekwerk) — "Vercel agent-desktop: computer use for coding agents through the accessibility tree", November 2026
- subagentic.ai — "Vercel Labs ships agent-desktop for coding agents", November 2026
- flaviocopes.com — "A deep dive into agent-desktop", updated November 4, 2026
- trend.ondo.sh — GitHub trend summary for vercel-labs/agent-desktop, November 2026
Frequently Asked Questions
Is agent-desktop an AI agent?
No. agent-desktop is infrastructure, not an agent — it has no model and no reasoning loop of its own. It is a Rust CLI plus a C library that an agent calls: every command prints one JSON envelope and exits. The observe-act-verify loop lives in your agent, whether that is Claude Code, Codex, Cursor, or a shell script.
How is agent-desktop different from screenshot-based computer use?
Screenshot-based computer use sends a picture of the screen to a vision model and asks it to guess pixel coordinates to click. The agent desktop vs screenshots debate comes down to one thing: structured data versus guessed pixels. agent-desktop skips the pixels entirely: it reads the operating system's accessibility tree and gives every button, field, and menu a stable ref, like a handle. Refs do not break when the window moves, the theme changes, or the display scales, and a stale ref fails with a clean error instead of a wrong click.
Which operating systems does agent-desktop support?
macOS 13 or later (Intel or Apple Silicon) via the Accessibility API, Windows 10 1809 or later (x64 or ARM64) via UI Automation, and Ubuntu 24.04 or later (x86_64 or aarch64, X11 or Wayland) via AT-SPI. macOS requires an Accessibility permission grant; Windows works at the terminal's integrity level with no grant; Linux needs the at-spi2-core package.
How much does agent-desktop cost?
It is free and open source under the Apache-2.0 license. You install it with npm, Homebrew, or a native binary from the GitHub releases page, and there is no account, subscription, or usage fee attached to the tool itself.
Can agent-desktop automate websites?
No — that is the one boundary to remember. agent-desktop drives native desktop apps; web pages are the job of its sibling tool, Vercel Labs' agent-browser. It is also invisible to apps that expose no accessibility information, such as some canvas-heavy apps, which still need screenshots or other methods alongside it.
