Apple builds web replicas to train visual agents

On May 7, 2026, Apple researchers introduced Weblica, a framework that turns web interactions into local, repeatable environments for training visual agents. Training an agent to navigate the web sounds straightforward until an experiment has to be repeated: pages change, forms expire, cookie notices appear, servers respond at different speeds, and bot-detection systems interrupt a route.

The project does not teach a model to read a page’s source code or rely on its DOM. It starts from screenshots and coordinate-based actions, much as a person would use a browser. Its goal is for an agent to complete multi-step flows—searching, filtering, filling in a form, or navigating tabs—without every run depending on the public web behaving in exactly the same way.

Two ways to replicate the web

Weblica combines two approaches. The first, Weblica-Cache, records the HTTP traffic of a browsing session and replays it later. The system normalizes parameters that change between sessions, such as timestamps or tokens, so playback can work even without a network connection. It retains for training only sessions that an agent can complete under those conditions.

The second, Weblica-Synth, creates interactive sites from scratch. The authors describe a pipeline in which coding agents generate HTML, CSS and JavaScript from a navigation capability, a site category and a visual style. The sites are self-contained; their states are kept during a session and they are validated with screenshots before joining the training set.

The purpose of the mix is practical. HTTP replicas bring in elements from real sites, while synthetic environments make it possible to produce many variants of an interaction without waiting for an external server. The paper reports retaining 15,600 cached environments and tasks, and generating 2,810 synthetic sites; 2,560 of the latter were reserved for training and covered 44,227 tasks.

An eight-billion-parameter agent

On these environments, the team post-trained Qwen3-VL models with supervised learning and reinforcement learning. Rewards for open-ended tasks come from GPT-4o acting as a judge, receiving the instruction, the actions and the resulting screenshots. The authors report 88% agreement between that judge and human evaluations—useful evidence, but also a reminder that not every measurement rests on programmatic checks.

Their main model, Weblica-8B, averages 35.6% across three live-web benchmarks with one 30-step attempt. The base Qwen3-VL-Instruct-8B reaches 24.8% under the same budget: an absolute gain of 10.8 points. With more attempts and steps, the score rises, but that improvement also uses more compute when the agent is deployed. Comparisons therefore need to consider both the success rate and the budget of steps and retries.

The manuscript also reports that, with 60 total steps, Weblica-8B exceeds the average of one compared open-weight baseline using 100 steps. That is a promising result for open visual-interface agents, not a demonstration that they can safely carry out any operation on the web without supervision.

What it still does not replicate

The authors are direct about the limits. A cached replica is a snapshot: it does not capture later changes to a website or the full complexity of a dynamic application. Synthetic sites cover navigation patterns, but a gap between simulation and reality remains. And the evaluations start from a fixed goal in a single turn, whereas everyday use includes user corrections, memory across sessions and goals that evolve.

That caution matters for assistants that might purchase, book or handle information. Weblica provides infrastructure for training and comparing agents without turning every test into a wager against the live web. The next test is how much of that skill remains when the site, context and user intent change.