Anthropic gives Claude 3.5 Sonnet control of a computer

Anthropic updated Claude 3.5 Sonnet today and introduced a feature that brings language models closer to practical automation: Computer Use, a beta tool that lets Claude view a screen, move the cursor, click and type in an interface.

This does not mean the model has magical access to the entire computer. The application sends it a screenshot, and Claude decides what action to take next: click a button, fill out a form or browse a website. The system repeats that cycle until it completes the task or hits an obstacle. It is a significant step because it allows models to use software that has no API, the technical channel applications normally use to communicate with one another.

An agent that works on the screen

Until now, an assistant such as Claude could draft an email, summarize a document or write code, but a person still had to transfer the result to the relevant application. With Computer Use, the model can carry out part of that sequence on its own.

Anthropic cites administrative task management, data entry between systems, software testing and web research as examples. For developers, the new capability makes it possible to build agents: programs that receive a broad objective and chain together actions to achieve it.

The tool is available in beta through Anthropic’s API. The company has also said it is coming to Amazon Bedrock and Google Cloud’s Vertex AI, two platforms that allow companies to use AI models within their technology infrastructure.

The word beta matters. The model may misinterpret a screen, click in the wrong place or lose context when a task takes a long time. In OSWorld, a benchmark that measures the ability to operate computers through real-world tasks, Claude 3.5 Sonnet scores 14.9%. That is nearly twice the previous best published result of 7.8%, but still far below the 72.4% achieved by people.

Sonnet update also improves coding

The new version of Claude 3.5 Sonnet replaces the model released in June. Anthropic says it has made particular gains in coding and in tasks that require following multistep instructions.

On SWE-bench Verified, an evaluation based on real issues from software projects, the model solves 49% of the problems. That does not mean it can replace an engineer: the benchmark measures whether it can propose a code change that passes a repository’s checks. Even so, it has become one of the most closely watched indicators for comparing models in software development.

Sonnet retains its API pricing of $3 per million input tokens and $15 per million output tokens. A token is a unit of text that models use to process and generate language; in Spanish, it does not correspond exactly to a word.

The computer-use feature adds an indirect cost: screenshots also have to be processed as input. The longer and more visual a task is, the more tokens it will consume. So, at least for now, it will make more sense for repetitive, high-value processes than for simple actions a person can complete in seconds.

Claude 3.5 Haiku targets fast, high-volume tasks

Anthropic also announced Claude 3.5 Haiku, a smaller and faster version of its model family. The company says it matches or outperforms Claude 3 Opus—its most capable model to date—on several evaluations, at a lower cost and with greater speed.

Haiku is designed for use cases where handling many requests with minimal delay matters: customer-service assistants, content classification, data extraction or help built into digital products. Anthropic expects to make it available to users by the end of October, priced at $1 per million input tokens and $5 per million output tokens.

The combination of a powerful model, a cheaper one and a tool for operating interfaces reveals the sector’s direction. AI companies are no longer competing only to produce better answers in a conversation; they want models to take part in complete workflows.

The problem is not just technical

Giving a model control of a cursor also multiplies the risks. A website, email or document may contain instructions designed to manipulate the agent and derail its task. This attack is known as prompt injection: the model mistakes external text for legitimate instructions from the user.

Anthropic recommends using Computer Use in controlled environments, limiting permissions and maintaining human oversight, especially when handling decisions with financial, legal or security consequences. It also advises against using the tool for now in operations involving sensitive information or irreversible actions.

The difference between an eye-catching demo and a reliable tool will lie precisely there. Moving a mouse is relatively easy; knowing when not to click a button, asking for confirmation and resisting malicious instructions will determine whether these agents can move beyond tightly bounded tests and into real business processes.