Having Sources Is Not Enough: How to Check an AI Answer Using RAG

A citation is not a verdict

An AI tool answers a question, displays links, and seems to have solved the reliability problem. Yet there is a distinction worth learning: a system finding a document does not mean that document proves the sentence it has just written. A source may address the same topic, be out of date, describe a different population, or even contradict the summary.

That distinction matters whenever an answer will guide a purchase, a professional decision, a lesson, a news item, or a technical conclusion. The point is not to distrust every answer with RAG by default. It is to stop outsourcing the final step: checking the relationship between a claim and its evidence.

The capability to take away: distinguish retrieval, attribution, and support, then use five questions to test whether a source supports the claim the AI assigns to it.

What RAG does — and does not do

RAG means retrieval-augmented generation: before or while writing, a system searches a collection of documents and places passages in its context. The 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks combined a model’s learned, parametric memory with a retrievable Wikipedia index. That is useful: it can bring identifiable information into an answer and lets a collection be updated without retraining the entire model.

But the mechanism has two separate steps. First, a retriever selects passages that look relevant to the question. Then a generator writes an answer from those passages and its own instructions. Neither step, on its own, guarantees that every sentence is justified. The retriever can bring back text that is close but insufficient; the generator can turn a cautious statement into a sweeping conclusion.

It helps to name four levels, from least to most demanding:

1. Retrieval: the system found a related document.
2. Attribution: it identifies the document or passage it says it used.
3. Support: that passage actually entails the specific claim, with the same scope and conditions.
4. Coverage: the sources support every material claim, including limits and exceptions.

Many products show the first two levels well. The user needs to test the last two. The academic AIS framework, “Attributable to Identified Sources”, was designed to ask whether an externally grounded generated statement can be verified in an identified source. It does not score whether a link looks polished at the end of a paragraph; it examines the relationship between statement and source.

The five-question protocol

Imagine the AI writes, “This technique reduces errors by 40%,” and adds a link. Before repeating it, run these five questions.

1. What is the exact claim?

Break the sentence apart: which technique, which error, compared with what, in which population, and for how long? “Reduces errors” is not the same as “improves a metric on a test set.” AI answers often pack several leaps into one smooth sentence. The more consequential the decision, the smaller the units you should check.

2. Is the source original and accessible?

Open the link. A press release may describe a study, but it does not replace the study; a blog may summarize a rule, but it does not replace the rule. Look for the paper, filing, technical documentation, dataset, or official page that is relevant. If only a secondary summary is available, label the assertion mentally: not yet verified at the source.

A respected domain is not enough either. A primary source may be partial if it belongs to the seller. That does not make it useless, but it changes what it can prove. It may establish what a company claims or how it configured a test; it may not establish that a result will generalize beyond it.

3. Does the passage say that, or merely something nearby?

This is the central test. Find the exact paragraph, table, or quotation, and compare subject, verb, number, date, and condition. A source saying “in our experiments” does not support “in practice.” “May help” does not support “eliminates.” A study of adults does not demonstrate an effect in children.

Read the sentence before and after as well. Limits often live there: an exclusion, a methodological warning, a licence condition. The 2022 GopherCite research is a useful reminder: its authors trained models to cite specific evidence and concluded that citation is only one part of trustworthiness; even a claim supported by evidence need not be true. A citation makes the test possible. It does not pass the test in advance.

4. Does it cover the whole answer?

An answer can have a solid citation for its opening sentence while leaving its important conclusion unsupported. Mark the material claims: numbers, causal claims, comparisons, recommendations, and predictions. Then ask which source supports each one.

A common warning pattern is one citation at the end of a long paragraph. It may support a single sentence, not the five before it. Another is three links that create an impression of strength while all come from the same note or repeat the same announcement. Three echoes are not three independent checks.

5. Does it still apply to this question?

Check the date, version, jurisdiction, and context. A 2023 guide may have been replaced; a US policy may not apply in Spain; documentation for an earlier release may not describe the current product. RAG is often good at finding related text, but it does not always know which text is current or which permission you have to use it. If the answer does not show a date or version, ask for one.

Use it without becoming a full-time auditor

You do not need to audit every sentence in a trivial query. Match the effort to the potential harm. For a recipe or brainstorming, treating the answer as a draft may be enough. For health, law, money, safety, research, or a decision affecting someone else, open the sources and check the claims that change the decision. If you cannot open them, do not turn the answer into certainty.

You can improve the prompt too. Ask: “Separate facts from inferences; cite the exact passage for each claim; state date and version; say what your sources do not cover.” This does not force the system to be right, but it makes gaps visible and gives you something verifiable rather than persuasive prose.

The habit to keep

The lesson is not “AI citations are fake.” It is more exact: a citation is a lead to a check, not the check itself. RAG removes an important obstacle by bringing documents into the answering moment. The human task remains: test whether the document supports the sentence, whether it covers the conclusion, and whether it still applies.

When the next interface promises answers “with sources,” do not ask only, “Does it have links?” Ask: Which claim does each one prove, in which passage, and with what limitation? That habit will still work when the model, search engine, or product name changes.