GPT

GPT stands for Generative Pre-trained Transformer: a family of language models introduced by OpenAI that learns from vast amounts of text to produce and complete natural language. GPT is not a single model but an approach and a lineage that has evolved across generations since 2018.

What it means: a decoder-only transformer

GPT builds on the Transformer architecture described by Vaswani and colleagues in «Attention Is All You Need» (2017), but uses only its decoder half. Unlike models such as BERT, which read text in both directions, a decoder-only model is autoregressive: it processes a sequence from left to right and predicts each token from the ones before it. That design, presented by Radford and co-authors in «Improving Language Understanding by Generative Pre-Training» (OpenAI, 2018), is what gives GPT its generative nature.

How it is trained: self-supervision, instructions and RLHF

GPT's pre-training is often called «unsupervised», but the precise term is self-supervised: the model learns to predict the next token over unlabeled text, and the labels are not set by a person but drawn from the text itself, the token that comes next. There is no manual annotation, yet there is an explicit learning signal, which is why «unsupervised» is a loose simplification. The original 2018 work paired that pre-training with task-specific supervised fine-tuning, in a semi-supervised scheme. From that base model, a second stage turns the system into an assistant: instruction tuning and reinforcement learning from human feedback (RLHF). Ouyang and colleagues formalized it in InstructGPT (2022), showing that aligning a model with user intent can matter more than sheer size.

From GPT-1 to today's family

The lineage advances by generations: GPT-1 in 2018, GPT-2 in 2019 («Language Models are Unsupervised Multitask Learners», 1.5 billion parameters) and GPT-3 in 2020 («Language Models are Few-Shot Learners», 175 billion parameters), followed by GPT-4 (2023) and later models. Each leap widened the scale and the capabilities, including multimodality, but no single version should be presented as «the latest and most powerful»: the family keeps evolving, and any coronation is obsolete almost at once. What is stable is not a specific model but the method (self-supervised pre-training over a decoder-only transformer, refined with instructions and RLHF) and the idea that one system trained to predict text can solve many tasks. Which generation is most capable today is an open, shifting question; describing GPT as a family and an approach, rather than a single product, is what keeps the definition current.