How to Use OpenGCUA: The Beginner's Guide (The Open Computer-Use Stack, First Clone, and the Real-Device Harness)

CodexFuse Team · 2026-11-07T09:27:30.212Z · 18 min read min read

How to Use OpenGCUA: The Beginner's Guide (The Open Computer-Use Stack, First Clone, and the Real-Device Harness)

What is OpenGCUA and how do you use it?

Learning how to use opengcua starts with a correction: OpenGCUA is a research stack, not a consumer app. It is Alibaba Tongyi-MAI's open-source stack for generalist computer-use agents, and it gives researchers a harness, released trajectories, a macOS real-device harness, and a cross-device SDK. You use it by cloning the repository, installing the driver, pointing it at a model, and giving it a task in plain language.

An opengcua setup starts with the Git beginner guide, because every step below begins with a clone. The rest of this guide follows the official documentation as fetched on 2026-11-07, and it covers the paper, the interactive lab, the first run, and the benchmark numbers.

1. Open the interactive lab and run one lab with no install.
2. Clone the repository from GitHub.
3. Create a Python 3.12 virtual environment with uv.
4. Install the SDK and harness packages in editable mode.
5. Install CuaDriver and grant the macOS permissions.
6. Copy the demo config to local.json and add your API key.
7. Run mac_agent.py and give the agent its first task.

This opengcua tutorial follows the official OpenGCUA documentation, the paper, and the interactive lab. As of November 2026, the stack is 23 days old, so expect fast-moving releases and check the repository before you trust any single version number.

What does a generalist computer-use agent actually do?

A generalist computer-use agent, or GCUA, is a system in which the model chooses what to observe, how to act, and how to organize its work across visual and programmatic interfaces at every step. That definition comes from the OpenGCUA paper, and it is the idea the whole stack is built around. Most agents emit one click at a time, while a GCUA can write code that drives the computer.

OpenGCUA rests on two design ideas. First, code execution lets the model write GUI operations as Python programs that combine accessibility-tree calls, app APIs, shell commands, and GUI actions, with loops, branches, and checks. Second, harness-to-model decision transfer moves choices such as when to take a screenshot, what task state to record, and stating each program's intent from fixed harness rules to the model itself.

The practical payoff is fewer steps and higher scores, which the benchmark section measures. The OpenClaw beginner guide covers a different kind of agent, a personal assistant that runs on your own machine, and the contrast is useful: OpenClaw is a product you live with, while OpenGCUA is a research stack you evaluate with.

Where do the official OpenGCUA sources live?

The opengcua github repository is Tongyi-MAI/OpenGCUA, created on 2026-10-15. It holds 593 stars, 52 forks, and 112 commits as of 2026-11-07, which works out to 11.4 stars per fork and 4.87 commits per day over its 23-day life. The repository is public, its default branch is main, and its license is Apache License 2.0.

The opengcua paper is arXiv 2610.22200, titled OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use. Fifteen authors from Alibaba Token Hub submitted it on 15 October 2026 in the cs.CV category, and the paper itself is CC-BY 4.0. The paper is the source for the benchmark claims in this guide.

The opengcua interactive lab runs at tongyi-mai.github.io/OpenGCUA, and it needs no install at all. It walks through seven labs, one question at a time, with simulations and real runs clearly labeled. The site also hosts demo videos and a trajectories explorer for the eight evaluated models.

What do you need before you start?

An opengcua setup starts with a Mac. The runnable harness is macOS-only in this release.

You need macOS 14 or newer on Apple Silicon, Python 3.11 to 3.14 with 3.12 recommended, uv from Astral, Rust and Cargo 1.89 or newer, and the Xcode Command Line Tools. You also need an LLM endpoint, a model name, and an API key.

The driver is CuaDriver, installed with the mu command, and it needs two macOS permissions before it can work. Grant CuaDriver Accessibility and Screen Recording in System Settings under Privacy and Security, then quit and reopen CuaDriver. Without those grants the agent cannot see the screen or drive the mouse and keyboard.

The Docker beginner guide is worth reading if you run supporting services in containers, but the official install is uv plus editable source packages, or prebuilt wheels and a DMG from GitHub Releases, and there is no Docker install path for OpenGCUA. Do not go looking for a compose file, because the README and QUICKSTART contain none.

How do you clone and install OpenGCUA?

This opengcua tutorial follows the official OpenGCUA documentation step by step, and every command below is quoted from the README and QUICKSTART as fetched on 2026-11-07. No commands were executed in a live testbed for this guide, so treat the official docs as the authority if a step changes. The install takes one terminal session and about ten commands.

git clone https://github.com/Tongyi-MAI/OpenGCUA.git
cd OpenGCUA
uv venv --python 3.12
source .venv/bin/activate
uv pip install -e ./packages/sdk -e ./packages/harness/runtime

The two editable installs give you the SDK and the harness runtime from source. Prebuilt SDK wheels do not require Rust, but building from source does, so keep the Rust toolchain installed if you take this path. The next commands install the native driver and the browser engine.

.venv/bin/mu setup macos
.venv/bin/mu doctor macos
.venv/bin/mu setup browser --engine chromium
.venv/bin/mu doctor browser --engine chromium

The mu setup command installs CuaDriver, and its daemon is included in the app bundle, so there is no separate serve command to run. The mu doctor commands check that the driver and the browser engine are healthy before you continue. If a doctor check fails, fix that step before moving on, because the agent cannot run without a working driver.

How do you point OpenGCUA at a model?

Copy the shipped demo config to a private file with cp packages/harness/configs/macos_demo.json packages/harness/configs/local.json. Never commit local.json, because it holds your API key and it is gitignored for exactly that reason. Edit the file so it names your provider, your model, and your key.

{
  "api_key": "YOUR_API_KEY",
  "model_name": "qwen3.8-flash",
  "llm_base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
  "llm_protocol": "openai"
}

The default config points at Alibaba DashScope with the qwen3.8-flash model, which is the cheapest way to try the stack. Three protocols are supported, and the table below lists them. Your key can also come from an environment variable instead of the file: ANTHROPIC_API_KEY, DASHSCOPE_API_KEY, OPENAI_API_KEY, or API_KEY.

Config values load in a fixed order: the MAC_AGENT_CONFIG environment variable first, then local.json, then macos_demo.json. Command-line flags override file values, so a flag always wins over the file. The GitHub beginner guide is useful here if you have never cloned a repository or managed a gitignored file before.

How do you run your first task?

Run the terminal agent with .venv/bin/python packages/harness/runtime/mac_agent.py, and add --ui plain if you prefer plain-text output. The agent starts, reads your config, and waits for a task in plain language. Give it one small task first, such as opening an app and reading a value, so you can watch how it observes and acts.

.venv/bin/python packages/harness/runtime/mac_agent.py
.venv/bin/python packages/harness/runtime/mac_agent.py --ui plain
bash packages/harness/runtime/start_demo.sh

The third command starts the demo studio at http://127.0.0.1:8800, which is a web view of the same agent. The native macOS app is the third way to run it: download OpenGCUA-macOS-arm64.dmg from the v0.3.1 release, drag it to Applications, and use the menu-bar assistant. The DMG is ad-hoc signed, not notarized, so macOS may warn you on first launch.

A quick sanity check proves the SDK imported cleanly before you run anything. Run python -c "import mai_ui; from mai_ui import Runtime; import qwen_intelligence; print('sdk ok')" and look for the sdk ok output. The SDK package is qwen-intelligence-sdk, not opengcua, so a search for an opengcua package on PyPI will come up empty.

The macOS harness can also use local SKILL.md workflows on demand. Type /skills to list them, /skill pdf-form to select the built-in PDF-form workflow, or /skill new my-workflow to create your own. Personal skills live in the Generalist skills folder under Library Application Support.

How do you try OpenGCUA without installing anything?

Open the interactive lab in a browser and click through the seven labs, one question at a time. Every demo starts with one button, and the details fold out below it. Simulations and real runs are labeled on the page, so you always know whether you are watching a scripted walkthrough or a recorded live run.

The seven labs cover positioning, benchmark evidence, the real-device harness, the cross-device SDK, the project roadmap, and building with the project. The site also hosts four demo videos: a background-work video at 1 minute 8 seconds, a four-workflows visa-form comparison at 1 minute 50 seconds, hotel research at 3 minutes 33 seconds, and a click sequence at 25 seconds. The trajectories explorer links the released prompts and the complete OSWorld-v2 execution trajectories of all eight models.

The lab is the fastest way to judge whether the stack fits your work, and it costs nothing. If the lab convinces you, the install in the previous sections is the next step. If it does not, you have lost ten minutes, not an afternoon.

What do the OSWorld benchmark numbers actually show?

The paper evaluates eight frontier models on 82 OSWorld-v2 tasks, and the headline result is a mean partial score of 45.3 rising to 64.9, a gain of 19.6 points. That is a 43.27 percent relative gain over batched GUI-only execution, and steps per task fall 53.0 percent on average. Output tokens per task fall 27.2 percent, and six of the eight models use fewer tokens.

The weakest models gain the most, which is the result that matters for beginners on a budget. GLM-5.3-Flash gains 35.2 points, which is 15.6 points above the mean gain, and DeepSeek-V4.1-Flash gains 29.8 points, which is 10.2 above the mean. The gap between the strongest and weakest models shrinks from 39.1 points to 18.0 points, a narrowing of 21.1 points or 54.0 percent.

GPT-6 Astra shows the step savings most clearly, dropping from 102 steps per task to 34, a cut of 68 steps or 66.67 percent. Code execution alone adds 6.1 points and cuts steps 14.8 percent against a GUI-plus-CLI agent with the same GUI and shell access. Against a hybrid GUI-plus-CLI agent, the full stack adds between 4.4 and 9.1 points across four models.

The budget view is the most practical number in the paper. Qwen3.8-Flash reaches 60.5 points at about 0.38 dollars per task, while GPT-6 Astra reaches 75.7 points at about 15.26 dollars per task. That is 40.16 times the cost for 15.2 more points, so the cheap model is the right default for learning and the expensive model is the right choice when a task must succeed.

The eight evaluated models are GPT-6 Astra, Claude Opus 5, Gemini 3.8 Flash, GLM-5.3-Flash, Qwen3.8-Max, Qwen3.8-Flash, Kimi K3, and DeepSeek-V4.1-Flash.

What happens on a real Mac?

The lab runs four real macOS tasks: workbook reasoning, PDF form filling, shopping research, and hotel comparison, one run each. GPT-6 Astra completes 4 of 4 tasks at 34.5 steps per task and about 18.29 dollars per task, while Qwen3.8-Flash completes 2 of 4 at 67.3 steps and about 0.12 dollars per task. The cheap model costs 152.42 times less and takes 32.8 more steps per task, which is 1.95 times as many.

The reasoning-token gap is the most striking number on the page. Qwen3.8-Max uses 19.4 times as many reported reasoning tokens as GPT-6 Astra, 50.2K against 2.6K per task, and still completes fewer tasks, 3 of 4 against 4 of 4.

It also takes 1.5 times the steps and 5.7 times the output tokens. More thinking does not always mean better doing.

The workbook case study shows why. GPT-6 Astra builds a reusable solver, checks it against seven worked examples, and fills all 125 answers.

Qwen3.8-Max covers 85 questions in fixed row ranges and leaves 40 empty, which is 68.0 percent covered and 32.0 percent empty. Four tasks characterize the evaluated configurations rather than establish a general model ranking, so read these numbers as a first look, not a leaderboard.

If the always-on machine idea appeals to you, read what is VPS hosting before you rent a box, although the Mac itself must stay awake while the agent runs. A hosted box cannot replace the Mac for the real-device harness, but it can host the model endpoints and services around it.

OpenGCUA vs OpenAI CUA vs OpenCUA: what is the difference?

Opengcua vs openai cua is the comparison most readers ask about first, and the answer is a category difference. OpenAI CUA is a closed proprietary model that powered the Operator agent, a hosted product that ran in OpenAI's own cloud browser. OpenGCUA is an open-source harness and SDK from Alibaba that works with any model, including OpenAI's own GPT-6 Astra, which the OpenGCUA paper evaluates.

Operator launched on 23 January 2025 as a US research preview for ChatGPT Pro at 200 dollars a month, was folded into ChatGPT Agent on 17 July 2025, and the standalone Operator experience was retired. Developers can still use the model through the computer-use-preview model in the Responses API. The first release ships no trained model, which is the sharpest line between the two projects: OpenAI sells you a trained model, while OpenGCUA gives you the harness to evaluate any model.

Opengcua vs opencua is a name collision between two different projects, and the paper trail keeps them apart. OpenCUA comes from XLANG Lab at the University of Hong Kong and Dark Side of the Moon, and it ships trained models, OpenCUA-7B, OpenCUA-32B, and OpenCUA-A3B, built on Qwen2.5-VL and KimiVL. Its OpenCUA-32B model reaches 34.8 percent average success on OSWorld-Verified, which its authors present as open-source state of the art against OpenAI CUA at 31.4 percent.

Two related names deserve one line each so you do not mix them in. The macOS driver inside OpenGCUA is CuaDriver from trycua, a separate open-source desktop-environment project. MAI-Gym is a separate companion project from the same Tongyi-MAI team, an evaluation and online reinforcement-learning gym, and its numbers are not OpenGCUA numbers.

What does OpenGCUA cost, and what are its limits?

OpenGCUA itself costs nothing, because the software is free and open-source under Apache License 2.0 and there is no hosted service or subscription. Your costs are whatever your own model provider charges, and the benchmark tables show the range: about 0.12 dollars per task for Qwen3.8-Flash against about 18.29 dollars for GPT-6 Astra. CodexFuse offers no OpenGCUA tool, and this guide promises none.

The limits are the ones a 23-day-old research stack carries. You bring your own frontier model and key, because no trained checkpoint ships with the first release.

The harness runs on a Mac in this release, with SDK adapters for other platforms still preliminary. The v0.3.1 release page shows no publication date, so this guide states the version without dating it. Star and fork counts are a point-in-time snapshot, and they move daily.

Community support is a WeChat group and a WeChat assistant listed in the README, and the Discord invite is still marked coming soon. The project leads are Hanzhang Zhou, Long Li, and Yue Wang, and the README acknowledges the earlier Qwen-UI-Agent work and the OSWorld-v2 evaluation. If you need a stable production tool today, this stack is not it yet, and the paper says so by calling the harness preliminary.

Seven mistakes beginners make with OpenGCUA

Most early failures with OpenGCUA come from expecting a product instead of a research stack. The seven mistakes below are the ones the official docs warn about, directly or by omission. Each one has a one-line fix.

Sources

The OpenGCUA repository at github.com/Tongyi-MAI/OpenGCUA, fetched 2026-11-07, is the source for the repo figures, the install commands, the requirements, and the release contents. The QUICKSTART in the same repository is the source for the provider protocols and the config load order. The paper arXiv 2610.22200, fetched 2026-11-07, is the source for the benchmark headline figures and the GCUA definition.

The interactive lab at tongyi-mai.github.io/OpenGCUA, fetched 2026-11-07, is the source for the seven labs, the demo videos, and the real-device tables. The OpenAI Operator system card is the source for the Operator launch and retirement dates.

The OpenCUA project page and DeepWiki comparison are the sources for the OpenCUA model figures. No Facebook promotion applies to this article.

repo age in days: 23
stars per fork: 11.4
commits per day: 4.87
stars per commit: 5.29
OSWorld gain points: 19.6
OSWorld relative gain pct: 43.27
GPT-6 Astra steps cut: 68
GPT-6 Astra steps pct cut: 66.67
gap shrink points: 21.1
gap shrink pct: 54.0
GLM above mean: 15.6
DeepSeek above mean: 10.2
Qwen3.8-Max hybrid gain: 6.8
DeepSeek hybrid gain: 6.8
GLM hybrid gain: 9.1
Claude hybrid gain: 4.4
budget cost ratio: 40.16
budget score gap: 15.2
real-device cost ratio Astra vs Flash: 152.42
real-device steps Flash vs Astra: 32.8
real-device steps ratio Flash/Astra: 1.95
workbook empty share pct: 32.0
workbook covered share pct: 68.0

Frequently Asked Questions

Is OpenGCUA free to use?

Yes. OpenGCUA is free and open-source under Apache License 2.0, and there is no hosted service or subscription to pay for.

Your only cost is whatever your own model provider charges for API calls. The default DashScope config with qwen3.8-flash is the cheapest way to start, at about 0.38 dollars per task in the paper's budget view.

Does OpenGCUA work on Windows or Linux?

The harness you can run today needs a Mac, and it asks for macOS 14 or newer on Apple Silicon. The SDK has adapters for macOS, Android, browsers, and Linux, but the quick start you can actually run today is the Mac path. Windows and Linux users can still read the paper, explore the trajectories, and try the interactive lab with no install.

Do I need an API key to use OpenGCUA?

Yes. OpenGCUA ships no model of its own, so you must bring an LLM endpoint, a model name, and an API key in local.json or an environment variable.

DashScope, OpenAI, and Anthropic endpoints are all supported through the three protocols. Without a key the agent has no brain, and no step in the install can substitute for one.

Does OpenGCUA include a trained AI model?

No. The first release ships the harness, the trajectories, and the SDK, and the trained GCUA model is the next release.

That is the sharpest difference from OpenCUA, which ships trained models today. Until the model release lands, you evaluate frontier models from other labs through the harness.

What is the difference between OpenGCUA and OpenCUA?

OpenGCUA is Alibaba Tongyi-MAI's open harness and SDK for evaluating computer-use agents, and it ships no trained model. OpenCUA is a different project from XLANG Lab and Dark Side of the Moon that ships trained models such as OpenCUA-32B. The names sound alike, the teams are different, and the first release of OpenGCUA cannot do what OpenCUA's models do.