Automatic summaries: how to measure coverage and faithfulness
A summary can be fluent while omitting the decisive point or inventing a relationship. Reliable evaluation separates coverage, faithfulness and usefulness, and traces each claim to its source passage.
On 23 October 2019, the first version of T5 was published. The work cast translation, question answering and summarisation in one format: input text, output text. This unification made it easier to adapt a model to document condensation, but it did not settle the decision that matters to readers. A summary may sound flawless while omitting the decisive fact, changing an attribution or adding a relationship absent from the original.
The useful capability is to separate three qualities that a demonstration tends to blend: fluency, coverage and faithfulness. The first asks whether the text reads well; the second whether it preserves necessary information; the third whether every claim is supported by the source. None guarantees the other two. A fourth axis, usefulness, depends on whom the summary is for and what they must do with it.
Extracting and rewriting are different operations
An extractive summary selects sentences or fragments from a document. It narrows the opportunity to introduce new wording, but it can sever context, join incompatible passages or produce an awkward sequence. An abstractive summary generates a new account, as a person might when combining several ideas. It can integrate information more effectively, while creating room for names, numbers, negations or causes not found in the source.
The Transformer architecture, introduced on 12 June 2017 in “Attention Is All You Need”, replaced recurrence and convolutions with attention in its main design. Attention relates positions in a sequence, which matters when the premise of a conclusion appeared pages earlier. The original paper primarily evaluated machine translation; by itself, it did not demonstrate faithful summarisation of long documents.
BERT, published in 2018, pretrained representations conditioned on left and right context. Its experiments covered language understanding, question answering and inference, but not generative summarisation. Those representations can be adapted to score sentences or analyse documents. Attributing a writing capability the paper did not test confuses a mechanism with an outcome.
The dataset also defines the task
T5 converted every task into a text-to-text transformation and evaluated summarisation on datasets including CNN/Daily Mail and XSum. XSum was designed for “extreme summarisation”: one sentence stating what a BBC news story is about. That objective is not equivalent to summarising a contract, medical record or meeting, where one omission can change an obligation or decision.
PEGASUS, presented at ICML 2020, pretrained an encoder-decoder by removing sentences deemed important and asking the model to generate them from the remainder. Its authors tested twelve tasks spanning news, science, email, patents and legislative bills. Breadth strengthens the evidence, but each benchmark retains its own definition of length, relevance and correct reference.
A score must therefore travel with its dataset, test split, output length, model version and decoding procedure. A system optimised to produce a headline may look incomplete when asked for an executive brief. A model prepared for news may not recognise which clause matters in a contract.
A long document requires system design
When a source exceeds the context window, the model cannot handle it as one input. Splitting it and summarising each section may duplicate facts, lose cross-section relationships or elevate a secondary sentence because it appeared in several intermediate summaries. A hierarchy of chunks, intermediate summaries and a final synthesis also propagates errors: an early omission is unavailable at the final stage.
The operational answer begins before generation. Define the audience, length, mandatory questions, terms that must be preserved and inferences that are forbidden. A clinical note, judgment and transcript do not share a notion of importance. Simply asking “summarise this” hides those choices inside the model and makes success impossible to judge.
Then retain a map between every output claim and the passage supporting it. The map does not automatically make the claim true: one fragment may contradict another or fail to justify the inference. It does lower review cost and allows direct statements, deductions and uncertainties to be labelled. In documents with stable pages or sections, those coordinates should accompany the summary.
ROUGE measures overlap, not truth
ROUGE, introduced in 2004, compares units such as n-grams or subsequences between a candidate summary and one or more reference texts. It enables reproducible evaluation of many outputs and can signal lexical coverage. It does not verify that an attribution is correct or that identical words play the same role. Two faithful summaries may use different vocabulary; an unfaithful one may repeat source terms.
The limitation was demonstrated in “On Faithfulness and Factuality in Abstractive Summarization.” Its authors conducted a large human evaluation of several systems and found unsupported content in all of them. Standard metrics correlated less effectively with faithfulness than textual-entailment measures. The result does not say every abstractive summary is false; it shows that fluency and overlap are insufficient verification.
FactCC approached the problem by training a verifier on artificial transformations and asking it to locate support and conflict spans. Human evaluation indicated that highlighted passages could assist review. It is an aid, not a final judge: training begins with synthesised error types, and performance differs between CNN/Daily Mail and XSum. An inconsistency detector also needs validation in its deployment domain.
Four tests instead of one overall score
The first test is coverage. Before seeing the output, a reviewer lists indispensable facts, decisions or clauses and then records which are present. This avoids confusing “resembles a reference summary” with “contains what this reader needs.”
The second is faithfulness. Every claim is checked against its passage, with dedicated review of entities, quantities, units, dates, negations, causal relationships and who said what. Omissions are classified too: losing an example is not equivalent to losing the condition that limits a figure.
The third measures writing quality without allowing it to offset errors: coherence, clarity, repetition and length. The fourth is extrinsic: representative users receive the summary and researchers check whether they complete the right task. A clinical summary can be measured by the information a professional retrieves; a meeting summary by correctly identified decisions, owners and deadlines.
Results should be reported by axis and document type. An overall average may conceal a system that works on short prose but fails on tables, appendices or conflicting sources. Originals, outputs, instructions, parameters and versions must also be retained. Changing the model, context window, segmentation or format requires revalidation: better style can coincide with worse faithfulness.
A protocol that survives the model
In operation, the interface should display the summary beside supporting passages, allow corrections and record disagreements. In high-impact settings, the system prepares a draft; the accountable person checks the source before making a decision. If a datum is absent or two sections disagree, the output should declare the gap instead of completing a probable story.
The competent question is not “which model summarises best?” but “best for which document, audience and cost of error, measured through which tests?” BERT, T5 and PEGASUS describe different mechanisms and regimes; ROUGE contributes one partial measure; FactCC another. Trust emerges when coverage, faithfulness, writing and usefulness are evaluated separately and every claim can return to the document supporting it.
This article was produced with artificial intelligence under human editorial oversight.
