Motorway cuts AI agent errors with continuous evaluation

Motorway, a UK used-car marketplace, has reduced incorrect results from its AI agent from one in every eight queries to one in 50. The improvement comes from an evaluation system developed with AWS that tests the agent before release and monitors its behaviour once it is in production.

The case matters because agents do more than generate an answer: they select tools, apply filters, query databases and maintain the context of a conversation. An error at any of those stages can return a car that does not meet the customer's requirements, even if the system's explanation sounds convincing.

A search tool that must understand imprecise requests

Motorway runs a daily auction in which up to 8,000 dealers bid on up to 2,500 vehicles. Its agent makes it possible to search the inventory using natural language, rather than forcing users to spend hours navigating CSV files and rigid filters.

The system exposes eight tools that combine structured filters across more than 89 attributes for each vehicle with semantic search. The latter converts the meaning of a query into numerical representations to find similar results even when the user does not use the exact terms in the catalogue.

For example, a query to find diesel SUVs under £25,000 near a dealership requires the system to interpret fuel type, vehicle type, budget and location. Another request for “something sporty and automatic for a family” requires it to decide which elements should be handled through filters and which through a broader search.

With around 1,500 concurrent users at peak times, the company cannot treat such failures as isolated cases. Selecting the wrong tool, passing incorrect parameters or losing a constraint introduced in an earlier turn can quickly erode dealer trust.

Three checks before the agent goes live

The architecture uses the Strands Agents SDK to build the agent and Amazon Bedrock AgentCore to run and monitor it. AWS describes AgentCore as a managed service for deploying and operating agents at scale.

Evaluation is split into two phases. During development and continuous integration, the team uses the open-source strands-agents-evals library to run test cases. After deployment, AgentCore Evaluations analyses real-world behaviour to identify errors that synthetic tests had not covered.

The controls are organised into three layers:

• Tool use: verifies that the agent selects the right tool and passes the correct parameters. The required threshold is above 95%.
• Reasoning: assesses whether the decisions leading to an answer are coherent. It must exceed 85%.
• Response quality: measures whether the final result is useful, accurate and actionable, with a minimum score of 90%.

The first two elements are not equivalent. An agent can show the user an apparently correct answer after following an unsound path. That behaviour tends to fail when conditions change or the query is phrased differently.

Deterministic evaluators are used to check tool use: code that compares the action taken with the expected action. Reasoning and output quality, by contrast, are reviewed by a language model acting as a judge. This approach, known as LLM-as-judge, provides flexibility for assessing open-ended responses, but it also introduces variability.

Repeating tests to measure consistency

Variability is a central issue for agents based on language models. The same request can produce different trajectories across separate runs. The system therefore proposes the pass^k metric: a test is run several times to measure whether the agent consistently meets the criterion, rather than passing only on a favourable attempt.

The combination of pre-release testing and production monitoring has also cut the time needed to detect incidents from a few hours to a few minutes, according to AWS and Motorway. The deployment pipeline includes five stages with quality gates that block a release if any metric falls below the set thresholds.

The result offers a practical lesson for companies looking to use agents in commercial processes: it is not enough to show that an assistant responds well in a demonstration. Companies need to verify which tools it activates, which data it uses, how it retains user constraints and whether it maintains performance when the same task is repeated. In services where a poor recommendation directly affects a sale, evaluation stops being a final stage and becomes part of the product.