What is Computer Use?
Also called computer control, GUI agent.
Computer use is a capability in which an AI model operates a graphical computer interface directly, viewing the screen as images and issuing mouse and keyboard actions. It lets a model work with any application, including software that exposes no API at all, at the cost of being slower and less reliable than calling a programmatic interface.
The loop is straightforward: capture a screenshot, send it to the model along with the goal and the history so far, receive an action such as a click at particular coordinates or a string to type, execute that action, then capture again. The model must locate its target visually and estimate coordinates, which is why small controls, scrolling, and dense screens are recurring sources of error.
Enormous amounts of work live in software that offers no automation surface, including internal tools, desktop applications, and legacy systems. Computer use is the universal fallback for those cases. It became a headline capability of frontier assistants during 2024 and 2025, and measured accuracy on realistic multi step desktop tasks remains well below what demonstrations suggest.
Each step costs an image and a round trip, so a task an API would finish in one request may take dozens of steps and several minutes. Errors also compound, because a misplaced click leads to a screen state the model did not expect. Practical systems keep tasks short, verify state between steps, and fall back to a programmatic interface wherever one exists.
This is among the highest risk agent capabilities, since the model can do anything the signed in user can do and everything it sees is untrusted input. A message displayed on screen can attempt to redirect it. Run it in a dedicated virtual machine or container with a separate account, limited credentials, and restricted network access, and require confirmation for irreversible actions.
Key points
- The model sees screenshots and issues mouse and keyboard actions.
- Universal fallback for software that exposes no API.
- Slow and error prone compared with programmatic calls.
- Errors compound across steps, so keep tasks short and verified.
- Run it isolated, with confirmation before destructive actions.
In practice
An expense workflow lives in a desktop application from the 1990s with no scripting support. An agent takes a screenshot, sees the entry form, clicks the vendor field, types the vendor name, tabs to the amount, types the figure, screenshots again to confirm the values landed in the right fields, then clicks save. Six actions and six screenshots for something an API would have done in one request.