An agent is not better just because it reflects more

On July 17, 2026, a research team released an unusual experiment on AI agents: rather than asking only how much information they extract, it examines how they behave while working. The preprint compares a fixed language-model workflow with agents that can use tools, retry a task, reflect on a failure and retrieve memories from earlier work. Its conclusion is restrained: those components substantially change execution, but they do not automatically produce a large improvement in final output.

The same job, two ways of doing it

The test asks systems to read scholarly PDFs and identify datasets mentioned in them. They must then create structured records with a dataset name, description, domain, paper reference and, when available, a link. Evidence may appear in an abstract, methods section, table, figure caption, reference list or associated code, making it a useful setting for studying multi-step decisions.

The fixed workflow always follows the same order: obtain a PDF, extract text, identify dataset names and fill in records. An agent can stop when an output looks incomplete, try another tool, consult memory or rerun a step with a different instruction. The comparison keeps the same corpus, output schema and model family, so differences can be linked to execution control rather than to a changed task.

Measuring decision traces

The authors call their target behavioural controllability. An agent, in their view, should be observable, configurable, reproducible and comparable. Observability means retaining traces of tools, intermediate results, retries and stopping decisions. Configurability means that retry limits, quality thresholds, memory and reflection are explicit controls rather than hidden features. Reproducibility requires recording models, prompts and run conditions.

This shifts attention away from a single score. Two systems may return the same number of records, while one reaches them through repeatable, inspectable steps and the other makes unnecessary calls, repeats errors or injects irrelevant memory. For an organisation that needs to audit information extraction, that difference can matter as much as raw coverage.

Reflection and memory have costs

The study distinguishes an S0 workflow, an agent with rule-based reflection, another with additional model-based reflection, and an S2 design with twelve atomic tools and dynamic selection. The optimized agent can search sections, tables and references, verify links and normalise records according to document state. That flexibility may recover evidence a rigid sequence misses.

More actions also mean more runtime, more traces to inspect and new failure modes. The authors report that model reflection changes process signals substantially while yielding only modest output gains under a fixed retry budget. Memory can help reuse experience, but it can also introduce stale or incorrect information unless it is filtered.

The caution of a controlled experiment

The experiments use NeurIPS 2024 papers, process at most 50 documents and cap output at five records per paper. They do not have exhaustive human annotation for every dataset mention. Record counts and field completeness are therefore execution indicators, not definitive measures of precision or recall.

That limit does not make the work unhelpful; it clarifies the question it answers. It does not prove that a reflective agent is universally better. It offers a more demanding way to evaluate one: ask not only for an answer, but also for evidence of how the system reached it and controls that allow its path to be repeated or corrected. For systems that act through tools, understanding the process may be a condition for justified trust in the result.