06-reference

technically computer use agents

2026-08-06·reference·source: Technically·by Tereza Tizkova

"How do 'computer use' agents work?" — Technically

Why this is in the vault

A structural taxonomy of computer-use agent architecture (screenshot-vision vs. accessibility-tree vs. hybrid routing) that gives RDCO precise vocabulary for a pattern its own harness already uses in miniature — routing each task to the cheapest reliable observation signal rather than defaulting to one method.

The core argument

Tizkova separates "computer use agent" into three things people conflate: a vision-language model (VLM) that reads a screenshot and clicks/types, a browser agent that only operates websites, and a true computer use agent that controls a full OS with persistent memory across sessions. Only the third is the real thing; the first two are components of it.

Three technical approaches to "seeing" and acting on a computer:

  1. Screenshot-based vision + click coordinates — Anthropic's original Computer Use (2024) and OpenAI's CUA. Expensive, high-latency (screenshot → inference → action loop), and fragile: UI reflow or a light/dark mode switch can break it.
  2. Accessibility tree / DOM parsing — reads the browser's structured tree of roles/names/states as text, far cheaper than images. WebVoyager benchmark: 40.1% accuracy text-only vs. 59.1% with vision added — structure alone isn't sufficient on dynamic, JS-heavy sites.
  3. Hybrid (the 2026 consensus) — either "default to structure, fall back to screenshots," or route each task to whichever backend fits. The author's employer, Factory (disclosed: "I work at Factory"), does the latter — four separate backends (accessibility tree for desktop apps, Chrome DevTools Protocol for web/Electron, a virtual PTY for terminals as text, and pixel capture only for full-screen TUIs). Only one of four backends works from pixels.

The piece frames every computer-use product as three stacked layers — model, harness (loop + tools + sub-agent orchestration), and computer (the execution environment, now converging on microVMs over containers) — and argues most arguments about what "counts" as computer use come from conflating these layers. It closes on open problems: action-space design is a legibility/coverage tradeoff (narrow = incomplete, broad = unauditable), and most production systems gate irreversible actions (deletion, submission, outbound messages) behind explicit confirmation.

Note: the Factory affiliation is self-disclosed once inline but the comparison section that follows describes Factory's own Droid Computers favorably relative to competitors — read that entry with the same discount any vendor-employee comparison piece deserves.

Mapping against Ray Data Co

The action-space tradeoff Tizkova names — "too narrow and the agent can't complete tasks that fall outside it, too broad and it becomes harder to audit what the agent actually did and why" — is exactly the design question behind RDCO's narrow-scoped critic sub-agents (verify-vault-write, verify-dispatch, station-critic): each is deliberately denied execution authority and given only the tools needed to read and judge, not act, so the audit trail stays legible even as the harness grows more autonomous. The article gives that already-lived design choice a named framework to reason from next time a new sub-agent type gets scoped.

The three-layer stack (model / harness / computer) also sharpens the 2026-07-21 harness-engineering note already in the vault: this piece adds the "computer" layer explicitly (execution environment, now microVM-consensus) as a third axis distinct from the harness itself, which the earlier note didn't isolate. That's a direct extension, not a restatement — RDCO's Claude Code sub-agents currently run on the host machine with no sandboxed "computer" layer of their own, which is the same gap the 2026-06-18 code-sandboxes note already flagged. Two independent Technically pieces now converge on the same open question for RDCO: should sub-agent execution move into an isolated environment, and this piece adds the vocabulary (accessibility-tree vs. pixel vs. hybrid routing) for what that environment would need to expose if it also had to drive a GUI, not just execute code.

Related