BERT, GPT, T5 and XLNet: evaluating classification and sentiment
The models start from different objectives and adaptation regimes. Reliability depends on labels, splits, metrics, domain, error cost and the ability to abstain.
On 11 October 2018, the paper introducing BERT was submitted. Its authors showed that a representation pretrained with left and right context could be adapted to language tasks by adding an output layer and fine-tuning. The advance improved text classification, but it did not settle the decisive question: which label does the system predict, for which population, and at what cost when it is wrong?
“Sentiment analysis” sounds like one task, yet it can mean the polarity of a review, an author's emotion, a stance towards an issue, the tone of a sentence or sentiment about a particular aspect. Positive, negative and neutral are not physical properties hidden inside text. They are categories defined by an annotation protocol.
The transferable skill is to separate architecture, adaptation regime and evidence. A model name describes how representations are learned or produced; reliability is demonstrated afterwards, through a defined label, suitable data, clean splits, metrics tied to harm and tests outside the comfortable domain.
The common foundation does not define the task
BERT and GPT belong to the Transformer family introduced in June 2017 by Attention Is All You Need. The paper proposed an attention-based sequence-transduction architecture without recurrence or convolution. Its main evidence came from machine translation and syntactic analysis; it did not itself establish that every Transformer classified sentiment well.
Attention allows a token's representation to incorporate information from other positions. This helps when a word changes meaning with its sentence or negation changes an adjective. It does not guarantee human-like understanding of irony, intent or emotion. Results still depend on the pretraining objective, adaptation data and output definition.
Architecture is the vehicle; a classifier is a wider system. It includes tokenisation, maximum length, truncation rule, examples, output head or prompt, threshold, model version and post-processing. Changing any one of them can change a decision while the commercial name stays the same.
BERT and GPT: representing or continuing
BERT pretrains bidirectional representations using left and right context. Sentiment classification commonly adds a head that receives a sequence representation and is fine-tuned on labelled examples. A controlled output over known classes is a practical advantage; the boundary it learns inherits the domain, noise and bias of its labels.
The first GPT was described in Improving Language Understanding by Generative Pre-Training. It pretrained an autoregressive model on unlabelled text and then fine-tuned it discriminatively for tasks including document classification. It was not simply a generator asked to supply a label: supervised fine-tuning was central to the method.
GPT‑3 explored zero, one or a few examples supplied in context without updating weights for each task. That regime reduces initial training needs but moves part of the system into the prompt: label order, demonstrations, instruction and output format must be versioned. Producing “positive” does not show that the model will preserve its classification under paraphrase, another language or a later month.
T5 and XLNet change the objective, not the need to evaluate
T5 expresses every task as text to text. Classification can become an input with a prefix followed by generation of a label string. The study compared objectives, architectures, corpora and transfer methods across numerous tasks. A common format makes infrastructure easier to share; it does not make the metrics or consequences of summarisation, translation and urgency detection equivalent.
XLNet uses a generalised autoregressive objective that maximises expected likelihood over permutations of the factorisation order. It does not feed the model a sentence with physically shuffled words. The paper sought bidirectional context without BERT's masking and compared the approaches on 20 tasks, including sentiment analysis.
Those results belong to particular datasets, configurations and dates. “Outperformed BERT” does not mean every XLNet defeats every BERT tuned for another language or domain. A reproducible claim preserves model version, test set, metric and procedure.
A label is a hypothesis about the text
Before selecting a model, each class needs a written meaning. “Negative” may mean that an author dislikes a product, reports a harmful event or uses words with negative valence. A story about falling inflation contains words of decline without necessarily expressing negative sentiment. A polite message may contain an urgent complaint.
The Stanford Sentiment Treebank was built to study composition and assigns labels to full sentences and to phrases in their parse trees. That supports analysis of negation and contrast; it also defines a domain: English film reviews. Performance there does not establish Spanish customer-service or financial-report performance.
Annotation instructions are part of the data. Teams should measure agreement between people and retain disagreement or uncertainty when a case is ambiguous. Forcing three reviewers into one label can erase exactly the signal a responsible system should treat cautiously.
Split data without letting the future leak in
Training teaches parameters; validation selects a model, threshold and configuration; testing estimates performance at the end. Repeatedly consulting the test set to guide changes turns it into another validation set and removes its value as a generalisation check. Duplicate or near-duplicate texts across splits produce apparent improvement through memory.
A random split does not always answer the operational question. To classify future messages, use a temporal split that trains on the past and tests on a later period. To measure transfer between companies or products, separate domains or authors. If the same reviewer's text appears on both sides, a model may learn that person's style rather than the target phenomenon.
In finance, a tone score does not establish an ability to predict prices. Evidence would require temporally ordered data, information available only up to each decision, transaction costs and an out-of-sample comparison. Leakage from a later date can turn impossible knowledge into a fictitiously successful strategy.
Accuracy is insufficient with imbalanced classes
If 95% of messages are ordinary and 5% urgent, always predicting “ordinary” reaches 95% accuracy while missing every urgent case. A confusion matrix shows which class is mistaken for which. Precision for a class asks how many alerts were correct; recall asks how many real cases were found.
Macro-F1 gives equal weight to each class and helps expose minority performance; a weighted average can be dominated by the common class. No metric chooses organisational cost. Missing an emergency may be worse than reviewing a false alarm, while incorrectly blocking a legitimate payment produces a different balance. Thresholds follow these costs and need review when prevalence changes.
Calibration adds another question: among cases given about 80% probability, is the system correct roughly eight times in ten? A score ranks examples; a calibrated probability supports review rules. Without testing it, displaying “98% confidence” can imply precision the evidence does not support.
Test language, domain and abstention
A useful suite contains negation, irony, intensifiers, quotations, bilingual messages, misspellings, jargon, long texts and topic changes. It also measures relevant subgroups and different time periods. The aim is not an impossible collection but discovering where error rates and error types change.
Abstention can turn an imperfect classifier into a useful component. If classes are close, context is missing or the input is outside the domain, the system sends the case for review instead of pretending certainty. Coverage — the share decided automatically — must be measured alongside quality on decided cases and workload sent to people.
After deployment, teams monitor input distribution, class prevalence, confidence, human corrections and later outcomes. A drop may come from new vocabulary, a campaign, another product or changes in annotators. Retraining without diagnosis can consolidate the failure.
A comparison somebody else can reproduce
A minimum report identifies model and version, objective, known pretraining data, adaptation method, prompt or head, labels, instructions, splits, duplicates, per-class metrics, calibration, threshold and error cost. It also preserves seeds or repeats where training or inference varies.
BERT may be a strong base for a narrow classification task with sufficient labels. An in-context GPT may accelerate a prototype when labels are scarce. T5 supplies a uniform interface; XLNet explores another pretraining objective. Those are design properties, not a permanent ranking.
Responsible selection does not begin by asking which name is largest. It begins by defining the decision and its harm, and ends with evidence on data representing actual use. Architecture, adaptation and evaluation are three separate columns; when a result omits one, it still describes potential capability rather than a reliable classifier.
This article was produced with artificial intelligence under human editorial oversight.
