An agent is not better just because it reflects more

On July 17, 2026, Lujia Zhang, Xingzhou Chen, and Hongwei Feng posted a preprint built around a more useful question than “does this agent get the answer right?”: can we reconstruct what it did, which controls constrained its behaviour, and why it retried or stopped? Their comparison of a fixed workflow and agents with reflection and memory extracts dataset names from scholarly papers. The result does not justify declaring a winner. It does teach a way to evaluate systems that chain models, tools, and decisions: a slightly longer output is not proof of a better process, and a longer trace is not proof that an agent reasoned better.

That distinction matters outside the lab. A company may ask an agent to review contracts, reconcile invoices, or assemble evidence in case files. If it measures only how many fields were filled, it may reward duplicates, fabrications, or expensive recovery loops. If it celebrates the fact that the system “reflects,” it may mistake activity for progress. The transferable skill is to separate three questions: what outcome did the system produce, what did it cost to produce it, and what observable evidence attributes an improvement to a particular decision?

What the preprint actually compared

The task is to read NeurIPS 2024 PDFs, detect the datasets they mention, and generate structured records. A system must identify a name, description, domain, paper reference and, when it can, a link and platform. Evidence may sit in the main text, a table, a figure caption, the references, or an external page. This is a plausible agent task because it requires several operations, but it also admits a conventional pipeline: download, extract text, request names, complete fields, and save JSON.

The authors completed three conditions over no more than 50 papers, with a cap of five records per paper. S0 was the fixed workflow. S1a added a controller, memory, retries, and rule-based reflection. S1b retained that harness and also enabled model-generated reflection. All three used the same openPangu-Embedded-7B model, corpus, schema, retry budget, and quality threshold. A fourth condition, S2, specifies twelve tools and dynamic planning, but the analysed package contained no completed S2 run. It is a design for follow-up work, not another result in the comparison.

In the archived runs, S0 generated 158 records from 42 papers; S1a generated 165 from 43; and S1b generated 168 from 43. Model reflection added three records over rule reflection and ten over the fixed workflow. The rate of discovered links did not improve monotonically: it was 19.0% for S0, 16.4% for S1a, and 18.5% for S1b. This is a useful antidote to the easy headline. More records may mean more coverage, but they may also mean more false positives; without a reference truth, both explanations remain open.

Outcome, process, and cost are separate axes

An outcome is measured against the task: precision, coverage, correct fields, and linked evidence. Process describes observable actions: tools invoked, errors returned, retries, states, and stopping decisions. Cost includes model calls, elapsed time, tokens, external operations, and human review. An agent can improve on one axis while becoming worse on the other two. An evaluation table should therefore expose them separately before a product decision combines them.

The paper reports 7,084 log lines for S0 and 50,189 in the agents’ shared log. That contrast proves that the agent harness emitted far more instrumentation; it does not prove that every line represents useful reasoning. In fact, S1a and S1b counters come from an accumulated log and cannot cleanly attribute all activity to an isolated condition. The authors also warn that traces reveal what the harness records, not faithful access to the model’s internal reasoning.

Interleaving actions with reasoning text did not begin with this study. ReAct proposed combining reasoning steps with actions that query an environment, while Reflexion added verbal feedback stored in episodic memory. These are architectures, not guarantees. They help when an action returns an informative signal and memory contains a valid lesson. When the verifier is weak, an agent can write a persuasive explanation of an error and preserve it for later repetition.

No reference truth means a different conclusion

The preprint’s decisive limitation is that it lacks a complete human annotation of every dataset present in the papers. Counting records and populated fields cannot yield precision or recall. An additional record is better only if it refers to a real dataset, does not confuse a method with a corpus, and links the right evidence. The study defines a future manual audit but does not yet report its results. It also has one run per condition, so the analysis is descriptive and estimates neither run-to-run variation nor statistical significance.

The right evaluation unit must be fixed before anyone inspects the output. Here it could be a paper–dataset pair verified by independent annotators. In contracts it might be a clause backed by a page and paragraph; in invoices, a line item linked to its document and accounting rule. Evaluators must then decide what counts as partial credit, how duplicates are handled, and which errors are most serious. Without that definition, “found more” is a metric that can be gamed.

A proxy should also be distinguished from the property it is meant to represent. Field completeness is a proxy for usefulness, but a populated field can be false. Retry count is a proxy for adaptation, but it may reveal a stall. A reflection score is a quality proxy only after validation against actual corrections. The core of NIST’s AI Risk Management Framework calls for documenting test sets, metrics, tools, uncertainty, and conditions similar to deployment; it also recommends independent assessment. For agents, that means connecting every counter to the risk or capability it supposedly measures.

How to isolate whether reflection adds value

A useful comparison changes one component at a time. The preprint moves in that direction by keeping the task and budget constant between S1a and S1b, but memory, retries, and rule reflection were already active in both. The three-record difference speaks only to enabling model reflection inside that harness in that run. It does not establish the general value of “being agentic,” nor can it reveal how much of the ten-record difference from S0 came from memory, retries, controllers, or prompt changes.

A stronger next experiment would run several seeds and separate components: fixed workflow; retries only; retries with rules; memory without reflection; reflection without memory; and combinations. Every condition needs the same model, tool versions, corpus, call limit, and time budget. Errors should be reported by document, not only as averages: a small mean gain may conceal extreme cost on difficult PDFs or a systematic failure class. Human evaluators should review records without knowing which system produced them.

Reflection should be triggered by verifiable evidence. Invalid JSON can activate a deterministic repair. A field without a citation can prompt a targeted search. Two attempts returning the same error can stop and escalate the task. By contrast, telling the model to “think more” without an external signal adds text and cost without defining success. The NIST measurement Playbook explicitly asks evaluators to test construct validity: whether a test measures the concept it claims to measure. For reflection, the construct is not the number of internal comments; it is the rate of errors corrected without creating new ones.

A minimum audit card for any agent

Before accepting a demonstration, write an audit card with six blocks. First, task and truth unit: what object is validated, and who annotates it? Second, controls: model, instructions, tools, corpus, temperature, limits, and versions. Third, outcomes: correct items, omissions, duplicates, and error severity. Fourth, process: external actions, failures, productive retries, and stopping reason. Fifth, cost: time, calls, tokens, and human minutes. Sixth, uncertainty: number of runs, variation, transfer limits, and missing data.

Then ask one small causal question: “With everything else held constant, does reflection correct more verifiable failures per added euro and minute?” If a trial establishes only that more traces exist, its conclusion is observability. If it produces more records without annotation, its conclusion is apparent coverage. If an independent audit confirms more correct records at an acceptable cost, then there is an operational improvement. Naming each level correctly prevents a promise from being sold as evidence.

The durable value of this preprint is not that it crowns an architecture. It shows why an agent needs two audits at once: one of the product it delivers and another of the observable route that produced it. Reflection deserves credit only when an isolated test connects it to corrected errors, better-grounded evidence, and acceptable cost. Knowing how to demand that connection will help readers evaluate the next “agent that thinks more,” whatever its name, model, or tools.