Nova Act can navigate a website; that still does not measure UX

On July 14, 2026, AWS published a reference architecture using Amazon Nova Act to generate journeys from documentation, execute them in parallel browsers, and analyse screenshots, timing, and traces. The design may expand what a team can observe between website releases. But it must not erase one boundary: an agent completing a test purchase does not prove that a person understands the interface, trusts it, or can use it with a disability.

The transferable skill is to separate three layers of evidence. The first asks whether the system functions: the final state and data are correct. The second asks whether the journey is operable: an agent finds a route repeatedly and leaves an intelligible trace. The third studies human experience: comprehension, expectations, accessibility, trust, and context. A browser agent supplies evidence for the first two and can generate hypotheses for the third, but it cannot represent users.

The correct date and what AWS presented

The original post is dated July 14, not July 17. It describes four layers. Product documentation in S3 feeds a Bedrock knowledge base; a Lambda function retrieves context and converts tasks into instructions at several levels of detail; DynamoDB, Lambda, ECS, and Fargate coordinate parallel runs; and Nova Act operates browser sessions. Results are saved, while another model calculates patterns, scores, and possible friction points for a dashboard.

The AWS sample repository exposes the vehicle for inspection: it records video, step-by-step HTML traces, and JSON calls, places results in S3, and aggregates statistics in DynamoDB. The solution does not contain the results of a user study or an empirical comparison with Selenium, Playwright, or human researchers. It is reference code for deploying a method, not evidence that the method measures usability by itself.

That precision changes how its dashboard should be used. A “discoverability score” computed from agent behaviour describes how that agent, with that model, instruction, seed, and website state, found a route. It may reveal a regression or suggest a problematic screen. It should not be presented as the share of people who will find the feature or as a validated satisfaction measure.

Design the oracle before launching the browser

Every test needs an oracle: an independent rule that decides whether the correct outcome occurred. “The agent said it finished” is a circular oracle. For a test purchase, checks should cover product, quantity, price, tax, test address, order identifier, and the absence of a real charge. For a preference change, inspect persisted state or an API rather than only the interface’s confirmation message.

Separate outcomes into states that explain failure. Functional success: the right state was reached. Product failure: the website returned the wrong result or blocked a valid path. Agent failure: the interface permitted completion, but the system chose badly. Infrastructure failure: the session, network, or dependency stopped. Indeterminate test: evidence is insufficient for attribution. Combining all of these as “failed” makes a tester error look like a product defect.

The Nova Act service card recommends that customers build evaluations tailored to their workflows and warns that the same prompt need not produce identical actions. AWS says its own scores are normally averaged across five runs. One product-team run is not enough either: repeat the task, preserve versions, and calculate success rate, step variation, time, recovery, and false failure by journey.

Goals, steps, and variations measure different things

A high-level instruction—“find a coffee maker and reach the order summary”—tests whether the agent discovers a route with little guidance. A detailed sequence—“open categories, apply this filter, select this product”—tests whether a prescribed path remains operational. A direct assertion checks one state. All three are useful, but they should not be aggregated as though they measured the same property.

Build a matrix of intentions and variations. Rows may include search, compare, register, recover password, purchase, and cancel. Columns may include desktop and mobile; new and returning sessions; language; slow connection; empty or extreme content; menu and search navigation; permissions and roles. Add equivalent wording changes to measure instruction sensitivity. Do not attempt every combination: select cases by frequency, impact, and areas changed in the release.

Documentation can seed cases, but should not be the only source of truth. If a manual omits a feature or prescribes an old path, the generator inherits the gap. AWS recommends a hybrid approach: generated baseline coverage plus manually defined flows for edge cases, new features, and scenarios requiring control. Complete that mix with real incidents, abandonment analytics, support queries, and user observations. The suite will then reflect the product in use, not only the product on paper.

A trace is not a person

The agent processes screenshots and chooses actions. A person arrives with prior experience, incomplete goals, cognitive load, time pressure, assistive devices, and cultural expectations. Agent success may conceal confusing copy because the model infers intent from visual patterns. Agent failure may exaggerate a difficulty that regular users solve from memory. Automation observes one class of interaction, not a population.

Accessibility provides an especially clear limit. The W3C’s Web Content Accessibility Guidelines 2.2 contain testable criteria for perceivability, operation, understanding, and robustness. A visual agent clicking a control does not thereby test its accessible name, focus order, keyboard navigation, error messages, or assistive-technology compatibility. Combine automated checks, expert review, and sessions with people who use those technologies.

To connect automation with UX, treat agent metrics as research signals. Many steps may suggest friction; repeated attempts on one control may justify inspection; divergence between vague and detailed instructions may point to poor discoverability. A person then reviews video, screenshot, and DOM, formulates a hypothesis, and checks it against behavioural data or user sessions. The agent helps decide where to look; it does not determine why a person abandoned a task.

The tester needs a perimeter too

An authenticated browser can buy, delete, or submit information. Use a test environment or separate accounts, synthetic data, test payment methods, and reproducible cleanup. Restrict destinations to allowed domains, block unnecessary files and tools, cap budget and steps, and define states that require stopping or escalation. AWS’s service card recommends domain allowlists, registering only tools needed by the workflow, and leaving file-path access blocked unless a task specifically requires it.

Instruction injection from webpage content is another risk. Text on a page can try to divert the agent from its task or persuade it to visit another site. Environment isolation reduces impact, but the oracle must detect actions outside the specification. Never use production credentials or authorise an irreversible transaction merely because the scenario is “only a test.”

Development should advance in stages. The Nova Act interface documentation separates exploration in a hosted browser, development and debugging with its SDK or extension, deployment through the CLI, and run monitoring in the console. Preserve that separation: first one controlled journey, then a repeatable suite, then continuous-integration execution, and finally monitoring for cost and failure. A generated test should not enter a release gate until its oracle and false-failure rate have been validated.

A report that supports a decision

For every journey, retain the objective, site version, initial data, model, instruction, seed when available, screenshots, actions, oracle result, and failure classification. The summary should expose repeated success, duration, step count, recovery, false positives, cost, and coverage by risk. Keep UX hypotheses separate from the human evidence that confirms or rejects them.

The AWS architecture is valuable because it turns journeys into reviewable material at scale. Its limit is just as instructive: automating an agent’s gaze does not automate an audience’s experience. If a team distinguishes function, operability, and human experience, Nova Act can reveal regressions and extend the map. If it mixes all three, a precise dashboard score can support a false conclusion about the people for whom the website was designed.