T5 (Text-to-Text Transfer Transformer)
T5, from Google (Raffel et al., 2019), treats every language task as a “text-to-text” problem. We explain that unified approach, its encoder-decoder architecture versus GPT and BERT, its pre-training on the C4 corpus, and variants such as mT5 and Flan-T5.
T5 (Text-to-Text Transfer Transformer) is a Google language model, presented by Colin Raffel and colleagues in 2019, that frames every language-processing task as a “text-to-text” problem: the input is text and the output is text, whatever the task. That unification makes it possible to use the same model, the same loss function and the same procedure to translate, summarize, classify or answer questions.
The “text-to-text” approach
The key is to prepend to the input an instruction indicating the task: for example, “translate English to German: …” or “summarize: …”. Even a classification task produces text, because the model generates the class word instead of emitting a numeric label. This removes the need to design a task-specific architecture: all tasks are treated as sequence generation.
Architecture and pre-training
T5 is an encoder-decoder Transformer, which distinguishes it from its contemporaries: GPT uses only the decoder and BERT only the encoder. It is pre-trained on C4 (the Colossal Clean Crawled Corpus), a large filtered web-text corpus introduced in the same work, with a span corruption objective: fragments of the text are masked and the model learns to reconstruct them, turning the pre-training itself into a text-to-text task. It was released in five sizes, from about 60 million to 11 billion parameters.
Variants and historical place
Widely used variants derive from T5: mT5, its multilingual version trained on more than a hundred languages, and Flan-T5, instruction-tuned on a large set of tasks to improve its performance on new ones. T5 is a milestone of 2019-2020, contemporary with the first wave of large transfer-learning models; it is not the latest thing, but it remains a solid, efficient reference for sequence-generation tasks such as summarization and translation.
This article was produced with artificial intelligence under human editorial oversight.
