What BERT changed and what reading in both directions really means
BERT learns contextual representations by masking words and using text on both sides. Its objective explains how an encoder differs from a generator and where its limits begin.
On October 11, 2018, four Google researchers released BERT, short for Bidirectional Encoder Representations from Transformers. Their original paper proposed pretraining an encoder to represent each piece of text using what appears on both its left and its right. That sentence captures the contribution, but it also creates two common misconceptions: BERT does not “read” like a person, and the original model was not designed to write long answers.
Its product is a contextual representation. In “she sat by the bank” and “she deposited money at the bank,” the same string receives different vectors because the surrounding words differ. Another layer, trained later for a particular task, can use that signal to classify a sentence, label an entity or locate the span that answers a question.
The useful skill is to examine three things whenever a model is said to “understand text”: which information it receives when computing each representation, which objective taught it, and how that representation becomes a measurable decision. That sequence prevents statistical context, a task score and human understanding from being collapsed into one claim.
The Transformer that BERT turned into an encoder
The Transformer architecture appeared in the 2017 paper Attention Is All You Need. That work replaced recurrence and convolution with attention mechanisms in a translation system containing an encoder and a decoder. BERT took the encoder side: it receives an available sequence and produces a representation for every position.
Self-attention calculates how strongly each token should influence another token's representation. It is not a causal explanation or a human-readable list of “important” words; it is a learned operation that mixes information within the sequence. Stacking layers allows BERT to build relationships across positions without having to process the text one word at a time like a recurrent network.
“Bidirectional” does not mean merely running one model forward and another backward and joining them at the end. Earlier systems such as ELMo already combined both directions. BERT's authors claimed joint conditioning on left and right context throughout every deep layer as the key change. For an analysis task, seeing the full sentence allows an ambiguous word's representation to change according to what comes before and after it.
How it learns without human-written labels
Using both sides creates a training problem: if a model can see the word it must predict, it can simply copy it. BERT addressed this with masked language modelling. Fifteen per cent of positions in each sequence were selected. Of those positions, 80% were replaced with the special [MASK] token, 10% with a random word and 10% were left unchanged. The objective was to recover the original token from context.
This distribution prevented the model from learning only how to react to a marker absent from normal use. The original version also included a second task, next sentence prediction: decide whether segment B genuinely followed segment A in the corpus or had been sampled at random. Both tasks generated labels automatically from raw text, so pretraining did not require people to classify billions of words by hand.
The English experiment combined BooksCorpus, containing roughly 800 million words, with English Wikipedia, containing roughly 2.5 billion. Those texts did not explicitly teach what a person, a correct answer or an offensive comment was. They taught the model to reduce error on its training objectives. Knowledge and bias in the representations follow from the available patterns and the learning procedure.
Tokens, positions and input limits
BERT does not receive words directly. The official implementation uses WordPiece, splitting terms into frequent and reusable components so an uncommon word can become several tokens. A sequence begins with [CLS], whose final representation often feeds classification tasks, and uses [SEP] to mark segment boundaries. Position and segment information are added to each token as well.
This tokenisation has practical consequences. A maximum length of 512 tokens does not mean 512 words and can be far too short for contracts, histories or full articles. Google's official repository notes that shorter sequences save memory and that extraction tasks must preserve alignment among characters, subwords and labels. Truncating a document without a strategy can place the answer outside the window.
The uncased variant lowercases text and strips accent markers before tokenisation. That can remove unhelpful variation in some tasks while destroying useful signals in others: “US” and “us,” proper names or accents. Choosing a checkpoint is not merely a leaderboard decision. Language, normalisation, vocabulary and length must match the actual data.
Pretrain once, fine-tune for each task
Google released BERT-Base with 12 layers, 768 hidden units, 12 attention heads and 110 million parameters, and BERT-Large with 24 layers, 1,024 hidden units, 16 heads and 340 million parameters. Pretraining was the expensive stage. A team could then add an output layer and fine-tune all parameters using labelled examples for its task.
The shape of that output changes with the problem. Review classification can use the [CLS] representation. Named-entity recognition predicts a label for each token while maintaining correspondence with subwords. Extractive question answering scores possible start and end positions in a passage. In the last case, BERT does not freely invent an answer; it selects a span from the input.
The paper reported then-leading results on eleven tasks, including GLUE, MultiNLI and SQuAD. The durable point is not to preserve those tables as a current ranking. It is the demonstration of a shared interface: general pretraining could transfer to multiple tasks with small changes to the output head. A score on one task does not automatically transfer to another; each fine-tune needs its own dataset, metric and validation.
Why BERT is not a generative chatbot
An autoregressive model in the GPT family learns to predict the next token and can continue a sequence from left to right. BERT sees an available sequence and reconstructs hidden positions; its original architecture is an encoder, not a causal decoder. It can fill a gap or supply representations to a classifier, but it does not naturally emit an indefinite paragraph one token after another.
The BERT Base model card states this boundary directly: the raw model supports masked language modelling or next-sentence prediction and is primarily meant for fine-tuning on sequence classification, token classification or question answering. For text generation it recommends a generative architecture. Calling BERT a “language model” describes its pretraining; it does not grant the same capabilities as a conversational assistant.
The recipe matters as much as the name
In 2019, the RoBERTa replication study showed that comparisons among methods were affected by data volume, training duration and hyperparameters. Its authors concluded that BERT had been substantially undertrained and that an optimised recipe could match or exceed later models. The lesson is not that every variant is better. It is that an apparent architecture gain may come from budget or protocol.
Comparing two encoders therefore requires keeping corpus, training steps, batch size, tokenisation, length, seeds and task fine-tuning visible. The base model must also be separated from the final system. A classifier built on BERT inherits its representations, but its performance depends on labels, thresholds, deployment distribution and validation quality.
Bias, time and what a score means
A checkpoint is not neutral because its corpus is not neutral. The BERT Base card demonstrates mask filling in which proposed occupations change when “man” is replaced with “woman.” This is an illustration, not an exhaustive inventory of harms. A hiring, moderation or health system needs task-specific tests across groups and error types; a global average can hide concentrated failures.
Domain and time shifts matter too. English books and Wikipedia do not automatically represent conversations, medical jargon, Spanish or recent documents. Fine-tuning can adapt a task, but it does not guarantee coverage of later changes or erase patterns learned during pretraining. Outputs must be tested on data close to deployment and tested again when that distribution changes.
A reading method that still works
For any text model, first determine whether it is an encoder, a decoder or a combination. Then open its objective: masking, next token, contrastive learning or another signal. Finally name the downstream task and its unit of measurement: classes, token labels, spans or generated text. Saying only that it “understands language” erases all three distinctions.
BERT mattered because it made a pretrained bidirectional encoder into a reusable foundation. Its legacy is not a claim of general understanding, but a method for producing contextual representations and adapting them. Once we know which context it sees, which loss it optimises and which head makes the decision, we can explain both success and failure without resorting to magical metaphors.
This article was produced with artificial intelligence under human editorial oversight.
