GPT

GPT names a family of language models that generates a sequence by predicting its next token from previous tokens. The initials combine three design choices: generation, pretraining, and the Transformer architecture. They do not identify one model or guarantee verified knowledge. A sentence can be grammatical, relevant, and false because generation rewards a likely continuation, not a check against the world.

This distinction prevents a misleading question: “What can GPT do?” The answer changes with the version, post-training, supplied context, connected tools, and product rules. A base model, a model tuned to follow instructions, and an assistant with search may share a lineage while behaving very differently. The useful skill is reconstructing which of those layers produced the result in front of you.

What the three letters mean

Generative means that the model produces a distribution over possible continuations. The system chooses a token, adds it to the context, and calculates the next one again. Repeating that cycle composes a long answer. A token is not necessarily a word: depending on the tokenizer, it may represent a whole word, a fragment, punctuation, or bytes. Visible length, token count, and computational cost are therefore different measures.

Pre-trained means that parameters are first optimized on a broad task before task-specific adaptation. The original 2018 paper, Improving Language Understanding by Generative Pre-Training, describes two stages: a language-modeling objective over unlabeled text followed by supervised fine-tuning for specific tasks. The work converted structured inputs into token sequences and used a decoder variant of the Transformer.

Transformer identifies the architecture family introduced in Attention Is All You Need. The 2017 paper proposed a network based on attention mechanisms, without recurrence or convolution. GPT uses decoder blocks with masked attention: each position can attend to the allowed context but not to future tokens it still has to predict. The architecture organizes computation; it does not turn a continuation into evidence.

From a token to an answer

Before generation, text is transformed into token identifiers. Each identifier enters as a vector and is combined with position information. Attention layers let one position use signals from other positions in the context; later layers produce scores over the vocabulary. Normalization turns those scores into probabilities, and a decoding rule chooses the continuation. Temperature, sampling, or choosing the maximum changes variability, not the knowledge stored in the parameters.

The context contains the instructions, examples, conversation, and documents that the system has placed inside the available window. Within one generation, GPT conditions its output on that sequence. Persistent memory between sessions, a vector database, or a search engine are external components that may supply new information. If a product “remembers” a fact or searches the web, attribute that capability to the complete system rather than assuming it appeared in the base model.

Pretraining compresses statistical regularities from the corpus into parameters. It does not create a searchable index linking every generated sentence to a specific page. The model may sometimes reproduce or recognize material; at other times it combines patterns and produces a plausible reference that does not exist. Asking for “the sources” after an answer does not establish where every claim came from. Verification requires opening real documents and checking locally that they support the claim.

Adaptation does not always mean retraining

For the first GPT, adaptation meant updating parameters with labeled task data. GPT‑3 demonstrated another route: specify a task through instructions and examples inside the context. The 2020 paper Language Models are Few-Shot Learners evaluated a 175-billion-parameter autoregressive model in zero-, one-, and few-shot settings without gradient updates during those tests. This is in-context learning: the input and observed behavior change, but the model weights do not.

Instruction tuning changes another layer. The 2022 work on InstructGPT began with GPT‑3, collected demonstrations written by labelers, fitted a supervised model, and then used human rankings to optimize it with reinforcement learning. On that study’s prompt distribution, evaluators preferred outputs from the 1.3-billion-parameter InstructGPT model to those from 175-billion-parameter GPT‑3. The result does not say the smaller model is universally more capable; it shows that size and fit to user intent are different axes.

A current application may add system instructions, document retrieval, code execution, tool calls, filters, and an interface deciding which history to send. Those components alter what users observe without necessarily changing the central model. “GPT answered” is therefore an incomplete attribution. Reproducing a result requires at least the model or snapshot, system message, conversation, tools, generation settings, and date.

Why fluency does not verify facts

Next-token prediction learns syntax, associations, formats, and many useful regularities because all of them help anticipate text. The same ability can confidently complete a false premise. Ask about “the Nobel Prize won by a person who never received one,” and the linguistic pattern may favor an explanation rather than a correction. Following the form of a question and checking its premise are separate capabilities.

A condition can disappear as well. An answer summarizes that a method “improves performance” while omitting that the result belonged to one dataset, metric, or configuration. The resulting text can preserve the conclusion while deleting the boundary that made it true. A number therefore needs a link beside the claim, and a quotation requires locating the words in the original. The model’s verbal confidence is not a calibrated measure of truth.

Connecting retrieval reduces one problem and introduces another. The system may receive the correct document but still has to choose the relevant passage, respect dates, and distinguish what the text states from what can be inferred. An answer with links is not automatically verified either: a link may concern another topic or support only part of the sentence. The cheapest test is local—open the source, find the datum, and read the surrounding scope and limitations.

“GPT” without a version is an incomplete claim

Product names can remain while the underlying model changes. The same label may acquire a new snapshot, more context, or different tools. Modalities are also easily confused: a service accepting images does not mean every model called GPT has that input; an interface browsing the web does not mean its parameters were updated with the page. A test record must identify the component evaluated and which external information it received.

Benchmarks do not remove that requirement. A result belongs to a version, example set, scoring method, and prompting protocol. It may measure closed-form answers rather than open use; it may accept several formulations or demand an exact match. Before carrying the number to another setting, ask whether the unit, success criterion, and available information are the same. “It beats the benchmark” does not mean “it solves every task with the same name.”

A test that separates the layers

To evaluate a capability, first write the task as an input, output, and criterion. Then prepare an ordinary case, one near the boundary, and one with a false premise. Record the full prompt, model, date, tools, and configuration. If the task uses documents, supply a known text and require every claim to identify its supporting passage. Check those passages outside the model. This separates generating an answer, retrieving evidence, and verifying it.

Next compare four conditions when available: base model, instruction-tuned model, model with in-context examples, and system with a tool. Do not assume the most complex condition wins. Apply the same criterion and retain abstentions, nonexistent citations, and omitted conditions. A tool may improve retrieval while worsening source selection; tuning may improve obedience while leaving factual errors. Naming the axis prevents a local improvement from becoming a universal capability.

When reading “GPT can do X,” ask: which version; model or product; parameter tuning or examples only; did it have tools; what context did it receive; how was it scored; and which failures occurred? If the answer does not let you reconstruct those conditions, the claim describes a demonstration, not a stable property of the family.

The transferable skill is separating architecture, training, adaptation, and system. GPT generates token by token with pretrained parameters; tuning changes its behavior, context can teach a task without changing weights, and tools add access to external resources. No layer replaces verification. Once version, objective, context, and evidence are fixed, an output is no longer judged by how convincing it sounds but by what it actually demonstrates.