Gemini 3.5 Flash integrates interface control for building agents

Google DeepMind announced on June 24, 2026 that Computer Use had become an integrated tool in Gemini 3.5 Flash. The official release says developers can build agents that observe, reason and act across browser, mobile and desktop interfaces through the Gemini API and enterprise agent platform. Integrated does not mean that the model receives unlimited control of a computer. An application still decides which actions to execute and in which environment.

Before the announcement, Google provided interface control through a separate model based on Gemini 2.5. Bringing it into Flash reduces the handoff between reasoning, tools and graphical operation. The company names continuous software testing and knowledge work among the uses, and shows demonstrations that analyze the Gemini app and inspect documentation for accessibility issues.

The capability is attractive because it reaches applications without a suitable API. If a person can complete a process by reading screens, selecting controls and filling fields, a visual agent can attempt the same route. That flexibility is also its weakness: the interface changes, contains untrusted text and combines information with actions that affect the world.

An agent is a loop, not a magic cursor

A Computer Use system has at least four parts. The environment produces a screenshot or representation of state. The model interprets that state and proposes an action. An application-side executor validates and performs the click, scroll or keyboard input. The new state is captured, and the cycle begins again.

The reference implementation linked by Google uses a software-controlled browser and offers a local Playwright option and a hosted Browserbase option. This exposes a crucial separation: the model chooses; developer code executes. A policy can sit between them to allow, modify, request confirmation or block an action.

Each part can fail differently. Perception may confuse two buttons. Planning may choose the wrong route. Coordinate translation may point to another element. State may change between screenshot and click. A correct action may have an unexpected effect because of permissions, session state or hidden data. Evaluating only the final result makes it impossible to identify which layer needs correction.

Why long tasks accumulate fragility

A multi-step task does not automatically inherit the success rate of one isolated step. If every transition has a probability of success below one, the probability of completing the entire chain without error decreases as transitions multiply. Some errors also change the state and contaminate later steps: typing in the wrong field can alter a search, purchase or recipient.

A UI-control benchmark is therefore not a reliability guarantee. The Gemini 3.5 Flash model card reports OSWorld-Verified and other agentic evaluations, but every score belongs to an environment, harness and criterion. Before transferring it to a company, ask how many steps the local task contains, which applications it uses and what a deviation costs.

The practical strategy is to add checkpoints. After a phase, the agent confirms an observable state: file created, total calculated, recipient selected or form still unsent. When the state does not match, the system stops or returns to a known point. This turns an opaque chain into segments that can be audited and recovered.

Prompt injection changes scale

An interface contains legitimate user instructions and content written by third parties. A page, document or message can include text that presents itself as an instruction to the agent. This is indirect prompt injection: untrusted data attempts to move up the hierarchy and direct actions. When a system only summarizes, the damage may be a false answer. When it can operate, the result can be data exposure, record modification or an unwanted communication.

Google reports targeted adversarial training for Computer Use and two optional enterprise safeguards: explicit confirmation for sensitive or irreversible actions and automatic stopping when an indirect injection is identified. The announcement also recommends sandboxing, human verification and strict access controls. The word “optional” matters. A builder must verify which layer is enabled instead of assuming that the model name configures the perimeter.

An injection detector should not be the only defense. Hostile text may not resemble known attacks, while legitimate content can trigger an alert. A secure architecture limits in advance which data the agent may read, which destinations it may reach and which operations it may execute. If an instruction passes the detector, it still encounters a bounded action radius.

The action contract before the first click

Before deployment, write a contract with five columns: action, object, condition, confirmation and reversal. “Read a public page” may proceed without a pause. “Create a draft” may be restricted to a temporary folder. “Send,” “delete,” “purchase” and “change permissions” should display the exact object and wait for a fresh human confirmation.

The contract must use least-privilege identities. An agent testing a website does not need an administrator account or personal email access. An invoicing agent may read a catalog and create drafts without being able to authorize payment. Credentials should be separated by environment and kept out of screenshots, logs and pages the model does not need to interpret.

Volume and time limits are also necessary. An instruction that appeared to create one file can enter a loop and produce hundreds. The executor should cap actions, allowed domains, duration, spending and transfer size. Reaching a limit ends the session and preserves state for review; it does not ask the model to work around the boundary.

How to test without risking production

The first trial uses a virtual machine, container or isolated browser profile, synthetic data and unprivileged accounts. Every run starts from the same state. The set should include normal tasks and designed obstacles: a popup, similar controls, a layout change, a hostile instruction inside a page and an action requiring confirmation.

Evaluation records more than completion. Count proposed and executed actions, correct confirmations, blocks, domain departures, recoveries, time and human corrections. A task completed after attempting a prohibited operation is not a clean success. A task stopped by a safeguard may be the correct result.

Review the trajectory step by step and assign every error to a layer: perception, planning, policy, execution or state change. This classification prevents answering every problem with a longer prompt. A coordinate error may need another controller; a permission error, a narrower identity; an intent error, a clearer instruction or approval point.

When to use an interface and when to use an API

Visual control is useful for closed applications and legacy processes, but a structured API provides fields, types, errors and less ambiguous operations. When a stable API exists for the critical action, Computer Use can be reserved for navigation, collection or bridges between systems. The interface then becomes an adapter instead of the only control layer.

Combining both routes lets a model read visual context while delivering a structured intent to a function that validates parameters. The application can show a preview and execute only after confirmation. This separation reduces the chance that a click at a changing position creates a different consequence from the intended one.

The transferable skill is to design an interface agent as a loop of perception, decision, policy, execution and verification. Gemini 3.5 Flash contributes the model and tool. The developer remains responsible for the executor, permissions, confirmations, limits and records. Useful autonomy does not remove those layers; it lets work advance while they preserve control.