GPT-2 and GPT-3: What Changes When a Text Predictor Scales
Objective, data, scale, context, decoding, and testing: a map for understanding GPT-2 and GPT-3 without confusing fluency with understanding.
Re-edited on July 30, 2026, this article corrects two figures and one inference. GPT-2 had up to 1.5 billion parameters and GPT-3 had 175 billion. Greater scale expanded fluency and learning from examples, but a convincing continuation does not demonstrate implicit understanding or authorize medical diagnosis.
The two models provide a durable case study: a system trained to predict the next token can reuse patterns for many tasks when those tasks are expressed as text. Judging that capacity requires separating objective, data, scale, context, decoding, and evaluation. A model name alone does not explain a result.
Autoregression: turning one sequence into many predictions
An autoregressive model factorizes the probability of a sequence by estimating each token conditioned on preceding tokens. During training, it sees the true prefix and learns to reduce next-token error. During generation, its own output becomes new context. An early error can redirect everything that follows even when every local step remains plausible.
GPT-2 and GPT-3 use decoder blocks inspired by the Transformer, with causal attention: each position can query only the permitted past. Multiple heads calculate relationships useful for prediction; they neither store a readable explanation nor guarantee that a head corresponds to a human concept.
Text is divided into tokens. One word may occupy one or several, and length varies across languages, names, and code. The model does not directly choose “an idea”; it assigns probability to a next token. Temperature, top-k, top-p, seed, length, and penalties turn that distribution into output. Comparing demonstrations requires matching these settings and counting attempts.
Training loss averages many tokens and can hide rare but important failures. During training, moreover, the model receives correct prefixes; during use it receives its own chain, including accumulated mistakes. Lower loss therefore does not fully describe repetition, contradiction, or drift in long passages. Complete sequences—not only average next-token probability—are evaluated, with length, genre, and language separated.
GPT-2: scale, WebText, and staged release
The report Language Models are Unsupervised Multitask Learners described GPT-2 variants up to 1.5 billion parameters and WebText, a corpus built from outbound links in Reddit posts that passed a voting threshold. It was neither “the entire web” nor a neutral set; the selection criterion incorporated that community’s preferences.
The study evaluated tasks without task-specific parameter updates by expressing them through context. Some results improved with size, while others remained far behind specialized systems. That combination is the finding: one loss can produce reusable representations, but transfer is measured task by task. Longer coherence does not establish correct knowledge.
The corpus also determines which world a model can reflect. Popular links do not represent private conversations, recent documentation, or every language. Quality filters may reduce noise while reinforcing a cultural definition of quality. A data record should include sources, dates, languages, licenses, deduplication, and excluded material. Without it, observers cannot tell whether a capability reflects generalization or coverage.
OpenAI initially staged the release of the largest model, citing concerns about harmful uses. Its report on release strategies and social impacts documented the process, partner work, and observations about generation. It helps distinguish a governance decision from a mathematical model property: risk, access, and evidence change with the use context.
GPT-3: scale and instructions as temporary data
The Language Models are Few-Shot Learners paper presented GPT-3 with 175 billion parameters and evaluated tasks with zero, one, and a few examples in context without updating weights. The model could infer a task format from a description or demonstrations. This is in-context learning; it is not necessarily a stable rule learned beyond that conversation.
Examples consume context, their order can matter, and an ambiguous demonstration can teach the wrong pattern. Capability tests repeat orders and formulations, compare zero- and few-shot settings, and report variation. A tuned baseline is also needed: solving a task without an update is interesting, but it does not automatically make the model the most accurate or efficient option.
Context acts as external, temporary working memory. It may contain rules, examples, and documents without necessarily changing parameters. When context disappears, so does that adaptation. A product records the complete template, limits which data enters, and tests sensitivity to conflicting examples. Saying a system “learned” without specifying whether weights changed or temporary conditioning occurred confuses two mechanisms.
Scaling laws for language models observed power-law relationships among loss, model size, data, and computation in the studied regime. An empirical curve does not promise that every capability will emerge when parameters are added. Growing a model without sufficient data or training budget misallocates resources, and lower loss does not translate uniformly across tasks.
Fluency, meaning, and truth require different tests
Predicting text requires learning syntax, associations, and regularities about the world expressed in a corpus. But fluent output and the human process of understanding are not the same variable. In Climbing towards NLU, Bender and Koller argued that training on linguistic form alone does not supply the relationship among expression, communicative intent, and world. This is a conceptual distinction, not a denial of every useful capability.
Factuality also requires its own evaluation. TruthfulQA built questions around imitative falsehoods and found that larger models could repeat false answers learned from human text. The result does not say scale always reduces truthfulness; it shows that linguistic probability and truth can diverge and that a conventional benchmark may miss the difference.
“Preliminary diagnosis” therefore cannot be attributed to a general model from text generation alone. Medicine requires a population, task, clinical comparison, sensitivity, specificity, calibration, harms, and oversight. The same applies to law, support, and journalism. Output may serve as a reviewed draft, search aid, or transformation; evidence defines its role, not style.
How to read a comparison of autoregressive models
First, record the objective and tokenization unit. Next, the data: origin, period, filters, duplicates, and possible test contamination. Third, the budget: parameters, training tokens, and compute. Fourth, the context: instruction, examples, order, and length. Fifth, decoding: algorithm, temperature, candidates, and selection. Only then compare metrics.
Testing should include accuracy, calibration, paraphrase robustness, subgroups, cost, and failures. For factual generation, split responses into claims and check sources; for classification, constrain labels; for code, run tests; for summaries, check faithfulness. Preference for writing style does not replace correctness.
Contamination must also be investigated. If a question or answer occurred in the corpus, the result measures something different from generalization to a new problem. Complete training data is not always known, so evaluations use dated sets, transformations, authorized private data, and uncertainty statements. A figure without traceability is presented as a limit rather than certainty.
Deployment adds one final layer: permissions and review. A model does not automatically execute what it can draft. Tools are limited by task, sensitive actions require confirmation, and interfaces show sources and uncertainty when available. Monitoring covers abstentions, corrections, and harms rather than satisfaction alone. A useful generator is one component inside an accountable process, not the whole process.
GPT-2 and GPT-3 were milestones because they showed how much transfer could emerge from autoregressive prediction when scale increased and context became an interface. The transferable skill is reconstructing any generative claim through objective, data, scale, context, decoding, and test. That framework recognizes progress without turning fluency into understanding or a possible use into clinical or business evidence.
This article was produced with artificial intelligence under human editorial oversight.
