T5 and BART: How to Compare Two Sequence-to-Sequence Models

Re-edited on July 30, 2026, this article preserves a useful comparison without turning it into a contest between brands: T5 and BART are encoder-decoder, sequence-to-sequence Transformer models, but they organize pretraining and task interfaces differently. That distinction helps explain how an objective produces the capabilities later measured.

Choosing between them is not a matter of saying that one “understands” while the other “generates.” Both read one sequence and produce another; both can be adapted for summarization, translation, answering, or textualized classification. The decision requires examining input corruption, reconstruction objective, task format, data, decoding, and evaluation.

The common base: encode an input and generate an output

The original Transformer used attention plus encoder and decoder blocks for translation. The encoder builds representations of the input; the decoder generates from left to right while attending to its prior output and those representations. The architecture enables parallel training over the input, but autoregressive generation still proceeds token by token.

Sequence-to-sequence does not mean that output has the same length or that the model copies. A question can produce an answer, a document a summary, and a sentence a label. The format unifies the interface, not the semantics: every task retains its data, errors, and criteria. Generating “positive” for sentiment is still classification.

RNNs and LSTMs already transformed sequences before these models. The change was not that they were incapable of handling long text, but that attention and parallelism altered the representation of dependencies and scaling of training. Context length, memory, cost, and data quality continue to impose limits.

T5: turning tasks into text to text

The T5 study examined objectives, architectures, datasets, and transfer within a common framework. Its main pretraining method corrupts spans of text and asks the model to generate the missing spans, represented by sentinel tokens. During fine-tuning, a textual prefix can distinguish tasks such as translation, summarization, or label production.

The durable contribution is methodological. When multiple tasks share an interface, differences in data and objectives can be controlled more carefully. A prefix does not grant capability by itself, however. Training must associate that instruction with examples, and a generated label can fall outside the permitted set. Production systems validate and, where appropriate, constrain output.

mT5 extended the approach to a multilingual corpus and evaluated cross-language transfer. “Multilingual” does not mean equivalent performance: text quantity and quality, tokenization, script, and task change coverage. Analysis belongs at the level of language and population rather than in an average that lets large languages hide smaller ones.

The text-to-text interface has a cost worth measuring. A one-word label may require decoding and permit spelling variants; a classifier with a closed head produces only allowed classes. In return, a generative format can share one model and examples across tasks. The choice is not aesthetic: compare invalid errors, calibration, latency, and maintenance. When output belongs to a catalog, an explicit constraint may be more reliable than hoping for linguistic compliance.

BART: learning to undo noise

BART was introduced as a denoising autoencoder for pretraining sequence-to-sequence models. It applies transformations such as masking spans or permuting sentences and trains the model to reconstruct the original text. The encoder can view the corrupted input bidirectionally; the decoder generates the reconstruction autoregressively.

That objective is intuitively close to tasks that transform one complete text into another, but it does not guarantee faithfulness. A summary can be fluent while adding an absent cause or number. The selected corruption also introduces bias: reconstructing shuffled sentences teaches a different regularity from recovering a masked name. Attributing improvement requires controlled comparisons of objectives with architecture, data, and budget held constant.

The idea of reconstruction pretraining did not belong to one family alone. MASS masked a sentence fragment and trained the decoder to predict it as a way to pretrain seq2seq models. Comparing these works teaches readers to examine the hidden unit, what the encoder sees, and what the decoder must produce instead of memorizing names.

Adaptation also changes the comparison. Updating every parameter, only part of the model, or lightweight modules changes memory, cost, and the risk of forgetting capabilities. More training epochs may improve the training set and harm new text. Preserve a time-based validation set, stop according to a prior criterion, and test the base model alongside the adapted one. A small task gain does not necessarily justify a large robustness loss.

Summarization and generation: decoding is part of the system

After training, output depends on greedy search, beam search, sampling, penalties, minimum and maximum length, and stopping criteria. Two runs of the same model can behave differently when decoding changes. A result selected from twenty candidates is not equivalent to a first attempt; a protocol should disclose the number of attempts and who selected.

Word overlap is insufficient for summarization. ROUGE was proposed as a package of automatic measures comparing candidate and human summaries. It can support comparisons, but high overlap does not guarantee faithfulness, and a correct paraphrase may overlap less. Claim, entity, number, and omission checks plus guided human evaluation are needed.

PEGASUS designed a pretraining objective for summarization by selecting and generating important sentences removed from a document. The example illustrates a general rule: moving a prior task closer to the final one may help, but it also makes checking for dataset-specific shortcuts more important.

Applications in poetry, journalism, law, or health do not follow from architecture. They require appropriately licensed corpora, a quality definition, accountable people, review, and domain testing. Generating a plausible article does not verify facts; summarizing a judgment does not determine its interpretation; producing empathetic text does not make a system a therapist.

A matrix for selection and evaluation

A comparison starts with six rows. Input: clean, noisy, multilingual, or structured text. Prior objective: what is hidden and reconstructed. Interface: prefix, task, and output vocabulary. Adaptation: data, modified parameters, and examples. Decoding: algorithm and constraints. Test: faithfulness, utility, subgroups, cost, and latency.

Next, create an identical baseline for both models. Split by document, author, or organization; equalize budget; test several seeds; and report variation. For classification, constrain labels and measure by class. For summarization, verify claims. For translation, evaluate terminology and phenomena by language. The winner may change with the task.

Error analysis uses categories that support action: omission, unsupported addition, wrong entity, altered number, contradiction, repetition, invalid output, and language failure. Count them by length and subgroup and preserve examples. Then change one variable—data, objective, decoding, or constraint—and rerun the test. Without that discipline, a higher average can conceal that the system merely learned to phrase the same error better.

Deployment records the model, adaptation set, instruction, decoding parameters, and input source. Users need to know when text is a proposal and what requires verification. Changes pass regression tests, and a previous version remains available for rollback. T5 and BART do not solve provenance, privacy, or accountability through architecture; those are properties of the service that integrates them.

Instruction tuning extends the interface but does not invalidate the analysis. FLAN demonstrated transfer after tuning models on many tasks expressed through instructions. Capability still depends on the mixture, templates, and relationship to the test. A natural-language instruction does not replace an output specification or evaluation.

T5 and BART matter because they make the connection among corruption, reconstruction, and transfer visible. The transferable skill is reconstructing any seq2seq model through input, objective, interface, adaptation, decoding, and test. That matrix supports evidence-based selection and prevents elegant text from being mistaken for faithfulness—or imagined application lists for observed results.