What RAG is: retrieving before answering does not guarantee accuracy

On May 22, 2020, a group of researchers presented a system that did something easy to explain and hard to execute well: before writing an answer, it searched for passages in an external collection. The paper that named retrieval-augmented generation, or RAG, combined two kinds of memory. One lived inside a model's parameters; the other was a searchable index of Wikipedia.

That pattern now sits behind assistants that answer questions about manuals, catalogues and internal documentation. Its advantage is not that it turns a model into an authority. It is more specific: it can place information selected at query time in front of the generator and let an operator change that collection without retraining the model. Its limit is equally specific: if the selection is poor, the answer can be less informed than it would have been with no retrieval at all.

The useful skill is to stop assessing a RAG system as one indivisible box. Ask three separate questions: did the evidence exist in the collection, did the search component retrieve it, and did the answer say only what that evidence supports? Each question points to a different failure and a different remedy.

Two memories and five steps

The original RAG work distinguished parametric from non-parametric memory. Parametric memory is what a model absorbed during training: patterns of language and knowledge compressed into its weights. Non-parametric memory is external text that people can inspect and replace. In the 2020 experiment, that external memory was a December 2018 Wikipedia snapshot split into roughly 21 million passages of one hundred words. The date matters: the system could consult its index, but the index could not contain later events.

A contemporary application usually follows five stages. Documents are ingested first: text must be extracted from PDFs, pages or records while preserving details such as date, version and permissions. The documents are then split into passages. A query is converted into a representation suitable for search. The retriever ranks candidate passages and gives a small set to the model. Finally, the generator writes using the question, instructions and retrieved context.

Preparation is not clerical trivia. A missing document cannot be retrieved. A poorly extracted table may lose the relationship between a number and its heading. A passage that is too short can separate an exception from the rule it qualifies; one that is too long can fill the context window with irrelevant material. A team must also decide which version wins when two policies conflict and which user is allowed to see each result.

How retrieval works: words, meaning and boundaries

A traditional search method such as BM25 rewards matching terms. A dense retriever represents questions and passages as vectors and compares them mathematically. The 2020 Dense Passage Retrieval paper, or DPR, showed that a dual-encoder system could beat a strong BM25 implementation on several open-domain question sets. Its headline result was an absolute improvement of 9 to 19 points in the chance that one of the top twenty passages contained the answer.

That result does not make semantic retrieval universally superior. In the same study, DPR trailed BM25 on SQuAD, a dataset in which questions and passages shared many words. The qualitative analysis also exposed complementary strengths: DPR captured lexical variation and semantic relationships, but it could miss rare, decisive phrases that literal term matching found. This is why some systems combine dense and lexical retrieval and add a separate reranking stage.

Similarity does not certify authority either. A passage can be close to the question while being outdated, opinionated or an exception rather than the rule. A retriever assigns relevance under its scoring function; it does not independently investigate whether the document is true. More results do not guarantee a better answer: each extra passage can add evidence, but it can also add noise and contradiction.

What the generator does with retrieved passages

The foundational paper tested two formulations. RAG-Sequence conditioned an entire output on the same retrieved documents; RAG-Token allowed different output tokens to depend on different documents. Current commercial products need not reproduce that architecture, but the distinction teaches a durable lesson: retrieval and generation are separate operations. A passage can reach the model correctly and still be summarized badly, blended with another passage or ignored.

The generator is not automatically confined to its sources. The original experiment recorded correct answers even when the answer did not literally occur in a retrieved passage because the model could draw on parametric memory. That flexibility helped on the benchmark, but it means we cannot assume that every sentence came from the index. Placing a link next to an answer is useful only when the linked text actually supports the specific claim.

This is the difference between provenance and support. Provenance says which documents the system saw. Support requires checking that those documents contain evidence for the sentence, that a lost negation does not reverse the meaning and that the answer did not add an absent detail. An interface can display polished citations while attributing a conclusion to a passage that never makes it.

The three places a RAG system fails

The first failure lies in the collection. The right document may be absent, a superseded version may remain, or extraction may have broken its contents. Diagnose this by searching directly for the evidence and checking metadata, dates and permissions. Retraining the language model does not fix a policy that never entered the index.

The second failure lies in retrieval. The evidence exists, but it does not appear among the candidates supplied to the generator. Inspect the actual passages and measure whether the first results contain the necessary support. Rewriting the query, combining lexical and dense search, changing passage boundaries or reranking candidates may help. Asking the generator to be more careful cannot recover text it never saw.

The third failure lies in generation. The right evidence arrived, but the answer contradicts it, extends it without support or hides its qualifications. This layer calls for claim-by-claim tests of faithfulness and correctness. Remedies may include better instructions, formats that explicitly bind claims to sources, a verification pass or abstention. A single end-to-end answer score cannot locate which of the three layers caused the error.

When the right answer is not to answer

Demonstrations usually test answerable questions: the evidence exists and the system must find it. That omits an everyday case, the question a knowledge base cannot resolve. The UAEval4RAG framework presented at ACL 2025 was built specifically to measure whether a RAG system rejects unanswerable requests. Its authors defined six types, from ambiguous questions and false premises to requests outside the database, and found that no single configuration performed best on both answerable and unanswerable questions across every dataset they tested.

The result does not support a universal number for all systems. The study used synthesized queries, tested single-turn interactions and acknowledged the need for more human-verified sources and application-specific evaluation. It does establish a product criterion: measuring only how often a system answers correctly rewards one that keeps answering without evidence. Evaluators must also test whether it asks for clarification, recognizes the boundary of its database or abstains.

Misleading retrieval can make an answer worse

RAG is often marketed as a treatment for hallucinations, but real corpora are not clean. They can contain mistakes, propaganda or texts that frame accurate details deceptively. A NeurIPS 2025 fact-checking study built a benchmark from verified claims and passages drawn from Reddit discussions. In that specific setting, every RAG model evaluated performed worse with potentially misleading context than its no-context baseline.

The conclusion has boundaries. GPT-4 assisted with intermediate labels, the task concerned political fact-checking, and the human study covered only 64 cases and four participants. It does not show that every RAG system always makes answers worse, nor that Reddit represents every corporate corpus. It shows something narrower: retrieval is not a neutral operation, and evaluations with ideal documents can overestimate robustness to conflicting or manipulated information.

An audit that works for any RAG system

Before trusting an answer, walk through the chain in order. First define the collection: which sources it contains, when they were updated, who can alter them and which access rules are enforced. Second inspect the passages that were actually retrieved, not merely the citations selected for display. Third split the answer into claims and determine whether each is supported, contradicted or unresolved by those passages.

Evaluation metrics should preserve the same separation. Retrieval can be measured by whether the evidence required for an answer appears among the first results. Generation can be measured for faithfulness to that evidence and correctness. The complete system can be measured through correct answers, genuinely supported citations and appropriate abstention. It also needs tests with stale documents, conflicting versions, ambiguous questions and misleading content—not only examples designed so that the answer is waiting to be found.

RAG is not a vaccine against error. It is an architecture that makes part of the path more visible and easier to update. Its real value emerges when that visibility is used for diagnosis. If the evidence was absent from the collection, external memory failed; if it was present but never reached the model, retrieval failed; if it arrived and the answer did not respect it, generation failed. Being able to distinguish those three breakdowns is far more useful than being able to expand the acronym.