Motorway measures its agent: tool selection rises from 87% to 98%
Motorway and AWS attribute improvements in its dealer search agent to layered evaluation. The case shows how to measure tools, trajectories, outputs and consistency separately.
On 23 July 2026, Motorway and AWS published results from an evaluation system for the agent that helps UK dealers search for vehicles. The most striking figure — moving from one incorrect query in eight to one in 50 — has a specific denominator: the Motorway and AWS account attributes the change to tool-selection accuracy rising from 87% to 98%.
This is not an independent audit. AWS sells part of the infrastructure, helped build the system and publishes the results with Motorway. It is nevertheless a technically detailed case that teaches something more durable than the improvement: an agent cannot be evaluated as a box that emits text. Tool choice, parameters, trajectory, final answer and consistency must be measured separately.
The transferable skill is to build a matrix in which each failure has its own metric, test set and threshold. A fluent answer can conceal a bad search; the right tool can receive wrong filters; one successful run can be a lucky exception. Measuring only the final text confuses these causes.
What Motorway's agent is trying to solve
Motorway runs a daily auction in which, according to figures published by the two organisations, as many as 8,000 dealers bid on up to 2,500 vehicles. Its agent converts natural-language requests into inventory searches. It exposes eight tools combining structured filters across more than 89 attributes with semantic search.
A request such as “diesel SUVs under £25,000 near my dealership” contains conditions that must become the correct fields and values. “Something sporty and automatic for a family” combines explicit filters with more open concepts. The agent decides whether to use structured search, hybrid search, distance, the dealer profile or another function, then constructs the parameters.
That path creates several opportunities for failure. It may choose semantic search when an exact filter was needed, omit the maximum price, lose a restriction introduced in an earlier turn, or accurately summarise results that were already wrong. With about 1,500 concurrent users at peak time, another figure from the case, a small percentage can become many failed experiences.
Layer one: tool, parameters and order
The first layer compares the observed trajectory with the expected one. For a structured request, a deterministic evaluator can verify the tool name, fields such as fuel type and price bounds, and call order when one operation depends on another. The open Strands Evals project organises tests into experiments, cases and evaluators, and supports expectations for outputs and trajectories.
Motorway set a threshold above 95% for this layer. That number is a decision in this case, not a universal constant. An inventory search may allow a user to correct a result; an agent that authorises payments or changes records requires stricter controls and perhaps deterministic prohibitions outside the model. The threshold should follow possible harm and recovery cost.
Tool selection and parameters should not be merged into one label either. Choosing the distance function is correct; sending another dealer's postcode is a distinct and potentially more serious failure. Separating the two shows whether tool descriptions, schema validation, permissions or context need to change.
Layers two and three: process and outcome
The second layer examines process coherence and uses a threshold above 85% in the case. The third asks whether the answer is helpful, accurate and actionable, with a threshold above 90%. Motorway and AWS use language models as judges for open questions while preferring deterministic checks for selection and parameters.
The two results belong together. An agent can reach a valid list after ignoring a constraint that happened not to alter the results. It can also follow a reasonable path and fail because inventory was stale. Process evidence helps diagnose; output evidence measures what the user received. Neither substitutes for the other.
A model-based judge provides flexibility but does not become ground truth. Its decision depends on the rubric, the context it sees and its own variability. Teams need human calibration, retained edge cases and disagreement monitoring. When an answer or trajectory is unambiguous, deterministic code offers a cheaper and reproducible signal.
Repeat: pass@k is not pass^k
Agents are non-deterministic: the same request can trigger different paths. The case separates two questions. pass@k measures the probability of at least one success across several attempts; it fits a task where finding one solution is enough. pass^k requires every consecutive attempt to succeed; it is closer to a customer expecting reliable results each time.
The gap grows quickly. At a 75% per-trial success rate, three consecutive successes have a theoretical probability of 0.75 × 0.75 × 0.75, about 42%, if trials are comparable and independent. A single passing result would hide that fragility. Real data can also be correlated — one troublesome phrasing may fail every time — so a report should include case count, repetitions, distribution and success criterion, not just an average.
The AWS reference repository implements three-layer evaluation and permits repeated trials per case. It is sample code, not a supported product or a production-ready recipe: its own notice requires testing, security work and adaptation before real workloads. That limitation matters as much as the architecture.
Before and after deployment are different problems
During development, a fixed case set can act as a gate: if a metric falls below its threshold, the build does not advance. Cases cover common paths, ambiguous language, multi-turn conversations, actions that should be refused and domain constraints. Motorway says its suite grew from 50 to 150 cases in three months as real failures became regression tests.
Production brings a distribution the laboratory does not know: rare terminology, concurrent load, incomplete data, latency and unexpected combinations. AgentCore Evaluations documents on-demand evaluation and online configurations that score sampled traces. In the described architecture, those signals feed metrics and alerts; they cannot retroactively block an answer a user has already received.
Shadow mode sits between the two: a candidate processes a copy of traffic without deciding what customers see. It can then move to a fraction of users and finally full deployment. The percentages and hours proposed by AWS are starting points for the example, not guarantees. Each organisation needs stop criteria, trace privacy controls and a rollback plan.
What improved, and what remains unknown
Motorway and AWS report several improvements together: tool selection from 87% to 98%, task completion from 82% to 96%, multi-turn context retention from 71% to 94%, monthly production incidents from 12 to two, and detection from an average four hours to a few minutes. The equivalence between 98% and one error in 50 follows their own definition of incorrect results.
The publication does not provide confidence intervals, the exact composition of the evaluation set, query volume in the before and after periods, or a controlled comparison isolating each change. No external reproduction is reported. The figures therefore demonstrate what the team measured under its protocol, not that the same pipeline will deliver the same gain for another agent.
A useful reading keeps both levels. The self-reported result makes the case worth examining; the method makes future agents easier to judge. Before accepting “accuracy improved”, ask accuracy of what, over how many cases, with how many repetitions, against which reference and with what consequence when it fails. That denominator turns an impressive demonstration into evidence that can be challenged and improved.
This article was produced with artificial intelligence under human editorial oversight.