agent-desktop Reaches 1.0 Under Vercel Labs, Giving Coding Agents Snapshot Refs to Native Apps Instead of Screenshots

A tool that lets AI agents operate desktop apps by reading their structure rather than their pixels has reached version 1.0 and changed hands. agent-desktop published 1.0.0 on October 28, 2026, at 06:45 IST, and 1.0.2 followed on October 31 at 02:47 IST. Between those releases the repository moved from its creator’s personal account, lahfir, into the vercel-labs organization on GitHub, and its README now carries a Vercel Labs product badge.

The project describes itself as “Computer use for AI agents: read and operate desktop apps through the OS accessibility tree.” It is licensed under Apache 2.0 and had about 4,300 GitHub stars by the morning of November 1.

A tool for agents, not an agent

agent-desktop is a command-line program and a C library for macOS, Windows and Linux. It contains no model. An agent calls it, reads the output and decides what to do next. “The loop stays in your agent,” the README says.

The basic cycle is observe, act, observe again. agent-desktop snapshot --app Finder -i returns a JSON document describing the app’s window as a tree of roles and names. Every element the agent can act on carries a ref, such as @s8f3k2p9:e8, and a list of available actions. The agent then runs a command like agent-desktop click @s8f3k2p9:e8 and takes a fresh snapshot to check the result. There is no pixel matching.

Using the operating system’s accessibility tree, the same interface screen readers rely on, means an agent works with named buttons, tree items and text fields instead of guessing coordinates from a screenshot. It also means the tool depends on how well each app exposes that tree.

Refs that expire on purpose

Much of the design is about not acting on stale information. Refs belong to the snapshot that produced them, so a bare @e8 without a snapshot id is rejected with INVALID_ARGS. When the interface changes underneath an agent, an action returns STALE_REF or AMBIGUOUS_TARGET, and the agent is expected to take a new snapshot. If several windows match, the error lists the candidate window ids.

On macOS, the CLI reuses a private command host that keeps native accessibility objects alive, so that a deleted row cannot quietly resolve to a different row with the same identifier. The host exits after 15 idle minutes.

Every command prints one JSON envelope with ok and either data or error. Errors carry a code, a disposition and usually a suggestion, and the README says to retry only when the disposition marks a retry as safe. For dense apps such as Slack, the docs recommend a --skeleton snapshot to map the window first, then a snapshot rooted at one region.

By default the tool is headless: ref actions go through the accessibility API without moving the pointer or stealing focus. A global --headed flag allows physical mouse and keyboard input.

Commands and platform coverage

The release lists 60 command names, 56 of them operational. Four low-level commands, key-down, key-up, mouse-down and mouse-up, refuse to run until a daemon exists that can release a held key or button. The rest cover observing, acting on refs, keyboard and mouse input, app and window management, notifications, the clipboard, waiting, session tracing and batching.

Support is broadest on macOS and Windows. Linux uses AT-SPI for the accessibility tree and needs the user’s graphical session with at-spi2-core. Mouse input there requires X11 and --headed, screenshots work for X11 windows or through the Wayland portal, and window control is X11-only. macOS needs Accessibility permission, and Windows needs no grant for apps at the terminal’s integrity level.

Installation is through npm, with native binaries for x64 and ARM64, or as release archives. Each release also ships libagent_desktop_ffi, so a program can call the library in process instead of spawning the CLI for every command.

Fitting into agent harnesses

The binary carries version-matched instructions for agents: agent-desktop skills get desktop prints the loop, ref, verification and retry rules, and skills get platform prints what differs on the current OS. The skill can be installed into a harness from skills.sh, or a two-line note can be added to a project’s AGENTS.md. The 1.0.2 release added a documentation site and full command reference.

What the accessibility tree exposes

The README warns that snapshots, clipboard reads, screenshots and debug output “can contain private desktop content” and should be kept out of public logs and uploads. An optional helper that sends screen descriptions to a model provider is separate from the native CLI.

For teams already running coding agents in a terminal, agent-desktop offers a way to reach native apps with structured, checkable actions. The project has not published an independent computer-use benchmark, so how it compares with screenshot-based agents on real tasks remains untested.