Weblica and the simulation-to-reality gap in web agents

On May 7, 2026, Apple researchers released Weblica, a system for training visual agents across thousands of website replicas. It addresses a fundamental tension: the live web offers diversity, but it changes, fails, detects bots and makes experiments difficult to repeat; a simulation is stable, but may teach habits that work only inside the mock-up. Weblica does not eliminate that tension. It makes the design measurable through two sources of environments — recorded HTTP traffic and generated sites — and reports results separately on synthetic and live-web evaluations.

Apple's primary research page summarizes a reproducible, scalable framework for visually navigating agents. The full Weblica manuscript supplies the detail: the agent receives a 1,280-by-720 screenshot and current URL, without a DOM or accessibility tree. It acts through coordinates for clicks and hovering, plus commands for typing, scrolling, tabs and stopping. The task therefore trains perception, planning and interface control together.

Replaying is not copying the whole web

Weblica-Cache records every HTTP request and response in a Playwright session. During replay, variable parameters such as timestamps or tokens would cause cache misses. The system compares recording and playback, identifies changing fields, generates domain-specific rules and creates synthetic responses for non-essential requests such as analytics. It then disconnects the network completely and keeps only sessions in which the agent completes the task.

The result is a replica of a route, not of the entire service. It can preserve content, dynamic layouts and observed interactions, but an unvisited route has no recorded responses. Normalizing a token may replay an equivalent request; it does not necessarily reproduce real authentication, changing inventory, fraud controls, permissions or effects in another system. The paper itself calls cached environments narrow and partial views: snapshots that do not follow later updates or capture the full complexity of dynamic applications.

The idea has a precedent. In 2017, World of Bits already used HTTP caching to build reproducible approximations of websites and train keyboard-and-mouse agents. Weblica automates rules, increases scale and combines that path with generation. The history clarifies the contribution: this is not “copying the Internet,” but a larger, less manual way to freeze verifiable parts of an experience.

The second web is built by models

Weblica-Synth starts from navigation capabilities. The authors had Qwen3-VL-32B attempt Online-Mind2Web tasks and used GPT-5.2 to analyze successful and failed trajectories. That produced 19,721 fine-grained capabilities grouped into 144 families, including navigation, forms, dates and maps. Each capability was combined with website categories and visual languages to keep every environment from converging on the same appearance.

Claude Code with Opus 4.5 generated framework-free HTML, CSS and JavaScript, plus at least ten tasks per site. The environments have no backend and keep session state with localStorage. The builder takes screenshots, tests functionality and iterates; a separate model generates images. The final set contains 310 sites targeting broad capabilities and 2,500 targeting fine ones. Of those 2,810 sites, 2,560 were assigned to training with 44,227 tasks and 250 to validation with 500 tasks.

Scale does not automatically equal effective diversity. Thousands of pages can share patterns because they come from the same builder, templates or vocabulary. Holding out 250 synthetic sites avoids evaluating on the same files, but they still belong to the same generative distribution. Weblica-val therefore measures generalization to new sites made by the Weblica process, not complete adaptation to the public web.

What the model learns and how it is rewarded

The team starts from Qwen3-VL-Instruct and combines supervised fine-tuning with reinforcement learning. Reinforcement learning draws 10,000 cached and 10,000 synthetic tasks. Some goals support exact verification; others are open-ended. For those, GPT-4o receives the instruction, action sequence and final screenshots, then decides whether the criteria were met. The authors report 88% agreement between this judge and human evaluations.

Agreement is not perfect accuracy, nor does it make the human an infallible reference. It means roughly one in eight compared decisions differed. If the judge awards systematic false positives, the agent may learn to satisfy visible signals without completing the intent. An audit should preserve disagreement examples, separate programmatically checked tasks from judged ones and test whether errors vary by domain, length or action type.

The final Weblica-8B model does not merely learn to point at elements. Visual-grounding evaluations remain close to the base model; the main gain appears in sequential behavior. That distinction matters because a correct isolated click and a completed task measure different abilities. Navigation requires retaining the objective, interpreting a changed state and deciding when to stop.

Read every score next to its budget

With one attempt and a 30-step limit, Weblica-8B raises the base Qwen3-VL-8B average from 24.8% to 35.6% across three live-web benchmarks, an absolute gain of 10.8 points. On the project's synthetic Weblica-val set, it rises from 56.9% to 70.6%. The gap between those figures signals the environment shift: the model succeeds more often on pages from the family used to construct training than outside it.

The study also increases test-time compute through more steps and parallel attempts. With 60 total steps, Weblica-8B reaches a 47.6% average on the live benchmarks, above the 42.4% achieved by the strongest compared open baseline with at least 100 steps. At 120 and 240 steps it reaches 58.9% and 68.9%. This scaling is useful, but pass@k asks, “Did any attempt work?” rather than, “How likely is the first attempt to work?” In a real service, retries consume time and may repeat an irreversible action.

A fair comparison needs at least five columns: first-attempt success, maximum actions per attempt, attempt count, total compute or latency and success criterion. If a score rises by multiplying executions, that gain is not free. It may be acceptable for search; for booking twice, publishing or paying, the strategy needs idempotency, verification and approval.

A live test has boundaries too

Online-Mind2Web, DeepShop and WebTailBench reduce the risk of measuring only inside the project's mock-up. Still, a live benchmark is a selection of domains, tasks and judges. It can change between runs and has its own failures. Repeating experiments three times and reporting deviations helps, but does not turn an average into a guarantee for any website.

VisualWebArena, another primary visual-agent benchmark, shows why disaggregation matters. Its 910 tasks span only three self-hosted environments and separate action difficulty, OCR, exact-image matching and visual inputs. In its initial evaluation, the best agent reached 16.4% versus 88.7% for humans, with OCR emerging as a bottleneck. This is not a direct comparison with Weblica, which uses different models and test sets; it is evidence that benchmark composition changes the average.

The useful question is not whether a model “can use the web,” but which combinations it handles: small text, changing menus, calendars, multiple tabs, cookie consent, authentication, slow loading, redirects or user corrections. A transferable evaluation labels tasks by capability and publishes failures rather than only one mean.

What Weblica leaves outside

The authors name three main limitations. Caches are partial and age; synthetic sites retain a gap with reality; and tasks are single-turn with a fixed goal. Everyday use includes conversation, evolving objectives, memory, preferences and corrections during the session. A navigation benchmark is also insufficient to justify high-risk operations.

Isolated environments are appropriate for learning without submitting forms, buying goods or changing accounts. That safety disappears when an agent connects to real services. A different authorization layer is required: minimum permissions, confirmation before external actions, spending limits, records and undo mechanisms. Knowing how to locate and press a button does not mean the system should decide to press it.

An audit of any simulation-to-reality claim can be reduced to six questions: which state is actually replicated; which routes are missing; how training and testing were generated and separated; who judges success; how much compute the score consumes; and which real capabilities remain outside. Weblica provides a measured improvement on its chosen benchmarks and a scalable environment-building method. Its most durable lesson is not that a replica is the web, but that every replica must state where it ends. Only then can a gain inside the simulator be interpreted without confusing reproducibility with reality.